code:
====================
device-mapper uevent
====================

The device-mapper uevent code adds the capability to device-mapper to create
and send kobject uevents (uevents).  Previously device-mapper events were only
available through the ioctl interface.  The advantage of the uevents interface
is the event contains environment attributes providing increased context for
the event avoiding the need to query the state of the device-mapper device after
the event is received.

There are two functions currently for device-mapper events.  The first function
listed creates the event and the second function sends the event(s)::

  void dm_path_uevent(enum dm_uevent_type event_type, struct dm_target *ti,
                      const char *path, unsigned nr_valid_paths)

  void dm_send_uevents(struct list_head *events, struct kobject *kobj)

The variables added to the uevent environment are:

Variable Name: DM_TARGET
------------------------
:Uevent Action(s): KOBJ_CHANGE
:Type: string
:Description:
:Value: Name of device-mapper target that generated the event.

Variable Name: DM_ACTION
------------------------
:Uevent Action(s): KOBJ_CHANGE
:Type: string
:Description:
:Value: Device-mapper specific action that caused the uevent action.
	PATH_FAILED - A path has failed;
	PATH_REINSTATED - A path has been reinstated.

Variable Name: DM_SEQNUM
------------------------
:Uevent Action(s): KOBJ_CHANGE
:Type: unsigned integer
:Description: A sequence number for this specific device-mapper device.
:Value: Valid unsigned integer range.

Variable Name: DM_PATH
----------------------
:Uevent Action(s): KOBJ_CHANGE
:Type: string
:Description: Major and minor number of the path device pertaining to this
	      event.
:Value: Path name in the form of "Major:Minor"

Variable Name: DM_NR_VALID_PATHS
--------------------------------
:Uevent Action(s): KOBJ_CHANGE
:Type: unsigned integer
:Description:
:Value: Valid unsigned integer range.

Variable Name: DM_NAME
----------------------
:Uevent Action(s): KOBJ_CHANGE
:Type: string
:Description: Name of the device-mapper device.
:Value: Name

Variable Name: DM_UUID
----------------------
:Uevent Action(s): KOBJ_CHANGE
:Type: string
:Description: UUID of the device-mapper device.
:Value: UUID. (Empty string if there isn't one.)

An example of the uevents generated as captured by udevmonitor is shown
below

1.) Path failure::

	UEVENT[1192521009.711215] change@/block/dm-3
	ACTION=change
	DEVPATH=/block/dm-3
	SUBSYSTEM=block
	DM_TARGET=multipath
	DM_ACTION=PATH_FAILED
	DM_SEQNUM=1
	DM_PATH=8:32
	DM_NR_VALID_PATHS=0
	DM_NAME=mpath2
	DM_UUID=mpath-35333333000002328
	MINOR=3
	MAJOR=253
	SEQNUM=1130

2.) Path reinstate::

	UEVENT[1192521132.989927] change@/block/dm-3
	ACTION=change
	DEVPATH=/block/dm-3
	SUBSYSTEM=block
	DM_TARGET=multipath
	DM_ACTION=PATH_REINSTATED
	DM_SEQNUM=2
	DM_PATH=8:32
	DM_NR_VALID_PATHS=1
	DM_NAME=mpath2
	DM_UUID=mpath-35333333000002328
	MINOR=3
	MAJOR=253
	SEQNUM=1131


code:
========
dm-zoned
========

The dm-zoned device mapper target exposes a zoned block device (ZBC and
ZAC compliant devices) as a regular block device without any write
pattern constraints. In effect, it implements a drive-managed zoned
block device which hides from the user (a file system or an application
doing raw block device accesses) the sequential write constraints of
host-managed zoned block devices and can mitigate the potential
device-side performance degradation due to excessive random writes on
host-aware zoned block devices.

For a more detailed description of the zoned block device models and
their constraints see (for SCSI devices):

https://www.t10.org/drafts.htm#ZBC_Family

and (for ATA devices):

http://www.t13.org/Documents/UploadedDocuments/docs2015/di537r05-Zoned_Device_ATA_Command_Set_ZAC.pdf

The dm-zoned implementation is simple and minimizes system overhead (CPU
and memory usage as well as storage capacity loss). For a 10TB
host-managed disk with 256 MB zones, dm-zoned memory usage per disk
instance is at most 4.5 MB and as little as 5 zones will be used
internally for storing metadata and performing reclaim operations.

dm-zoned target devices are formatted and checked using the dmzadm
utility available at:

https://github.com/hgst/dm-zoned-tools

Algorithm
=========

dm-zoned implements an on-disk buffering scheme to handle non-sequential
write accesses to the sequential zones of a zoned block device.
Conventional zones are used for caching as well as for storing internal
metadata. It can also use a regular block device together with the zoned
block device; in that case the regular block device will be split logically
in zones with the same size as the zoned block device. These zones will be
placed in front of the zones from the zoned block device and will be handled
just like conventional zones.

The zones of the device(s) are separated into 2 types:

1) Metadata zones: these are conventional zones used to store metadata.
Metadata zones are not reported as useable capacity to the user.

2) Data zones: all remaining zones, the vast majority of which will be
sequential zones used exclusively to store user data. The conventional
zones of the device may be used also for buffering user random writes.
Data in these zones may be directly mapped to the conventional zone, but
later moved to a sequential zone so that the conventional zone can be
reused for buffering incoming random writes.

dm-zoned exposes a logical device with a sector size of 4096 bytes,
irrespective of the physical sector size of the backend zoned block
device being used. This allows reducing the amount of metadata needed to
manage valid blocks (blocks written).

The on-disk metadata format is as follows:

1) The first block of the first conventional zone found contains the
super block which describes the on disk amount and position of metadata
blocks.

2) Following the super block, a set of blocks is used to describe the
mapping of the logical device blocks. The mapping is done per chunk of
blocks, with the chunk size equal to the zoned block device size. The
mapping table is indexed by chunk number and each mapping entry
indicates the zone number of the device storing the chunk of data. Each
mapping entry may also indicate if the zone number of a conventional
zone used to buffer random modification to the data zone.

3) A set of blocks used to store bitmaps indicating the validity of
blocks in the data zones follows the mapping table. A valid block is
defined as a block that was written and not discarded. For a buffered
data chunk, a block is always valid only in the data zone mapping the
chunk or in the buffer zone of the chunk.

For a logical chunk mapped to a conventional zone, all write operations
are processed by directly writing to the zone. If the mapping zone is a
sequential zone, the write operation is processed directly only if the
write offset within the logical chunk is equal to the write pointer
offset within of the sequential data zone (i.e. the write operation is
aligned on the zone write pointer). Otherwise, write operations are
processed indirectly using a buffer zone. In that case, an unused
conventional zone is allocated and assigned to the chunk being
accessed. Writing a block to the buffer zone of a chunk will
automatically invalidate the same block in the sequential zone mapping
the chunk. If all blocks of the sequential zone become invalid, the zone
is freed and the chunk buffer zone becomes the primary zone mapping the
chunk, resulting in native random write performance similar to a regular
block device.

Read operations are processed according to the block validity
information provided by the bitmaps. Valid blocks are read either from
the sequential zone mapping a chunk, or if the chunk is buffered, from
the buffer zone assigned. If the accessed chunk has no mapping, or the
accessed blocks are invalid, the read buffer is zeroed and the read
operation terminated.

After some time, the limited number of conventional zones available may
be exhausted (all used to map chunks or buffer sequential zones) and
unaligned writes to unbuffered chunks become impossible. To avoid this
situation, a reclaim process regularly scans used conventional zones and
tries to reclaim the least recently used zones by copying the valid
blocks of the buffer zone to a free sequential zone. Once the copy
completes, the chunk mapping is updated to point to the sequential zone
and the buffer zone freed for reuse.

Metadata Protection
===================

To protect metadata against corruption in case of sudden power loss or
system crash, 2 sets of metadata zones are used. One set, the primary
set, is used as the main metadata region, while the secondary set is
used as a staging area. Modified metadata is first written to the
secondary set and validated by updating the super block in the secondary
set, a generation counter is used to indicate that this set contains the
newest metadata. Once this operation completes, in place of metadata
block updates can be done in the primary metadata set. This ensures that
one of the set is always consistent (all modifications committed or none
at all). Flush operations are used as a commit point. Upon reception of
a flush request, metadata modification activity is temporarily blocked
(for both incoming BIO processing and reclaim process) and all dirty
metadata blocks are staged and updated. Normal operation is then
resumed. Flushing metadata thus only temporarily delays write and
discard requests. Read requests can be processed concurrently while
metadata flush is being executed.

If a regular device is used in conjunction with the zoned block device,
a third set of metadata (without the zone bitmaps) is written to the
start of the zoned block device. This metadata has a generation counter of
'0' and will never be updated during normal operation; it just serves for
identification purposes. The first and second copy of the metadata
are located at the start of the regular block device.

Usage
=====

A zoned block device must first be formatted using the dmzadm tool. This
will analyze the device zone configuration, determine where to place the
metadata sets on the device and initialize the metadata sets.

Ex::

	dmzadm --format /dev/sdxx

If two drives are to be used, both devices must be specified, with the
regular block device as the first device.

Ex::

	dmzadm --format /dev/sdxx /dev/sdyy

Formatted device(s) can be started with the dmzadm utility, too.:

Ex::

	dmzadm --start /dev/sdxx /dev/sdyy

Information about the internal layout and current usage of the zones can
be obtained with the 'status' callback from dmsetup:

Ex::

	dmsetup status /dev/dm-X

will return a line

	0 <size> zoned <nr_zones> zones <nr_unmap_rnd>/<nr_rnd> random <nr_unmap_seq>/<nr_seq> sequential

where <nr_zones> is the total number of zones, <nr_unmap_rnd> is the number
of unmapped (ie free) random zones, <nr_rnd> the total number of zones,
<nr_unmap_seq> the number of unmapped sequential zones, and <nr_seq> the
total number of sequential zones.

Normally the reclaim process will be started once there are less than 50
percent free random zones. In order to start the reclaim process manually
even before reaching this threshold the 'dmsetup message' function can be
used:

Ex::

	dmsetup message /dev/dm-X 0 reclaim

will start the reclaim process and random zones will be moved to sequential
zones.

repo_name: angery/linux


code:
======
dm-era
======

Introduction
============

dm-era is a target that behaves similar to the linear target.  In
addition it keeps track of which blocks were written within a user
defined period of time called an 'era'.  Each era target instance
maintains the current era as a monotonically increasing 32-bit
counter.

Use cases include tracking changed blocks for backup software, and
partially invalidating the contents of a cache to restore cache
coherency after rolling back a vendor snapshot.

Constructor
===========

era <metadata dev> <origin dev> <block size>

 ================ ======================================================
 metadata dev     fast device holding the persistent metadata
 origin dev	  device holding data blocks that may change
 block size       block size of origin data device, granularity that is
		  tracked by the target
 ================ ======================================================

Messages
========

None of the dm messages take any arguments.

checkpoint
----------

Possibly move to a new era.  You shouldn't assume the era has
incremented.  After sending this message, you should check the
current era via the status line.

take_metadata_snap
------------------

Create a clone of the metadata, to allow a userland process to read it.

drop_metadata_snap
------------------

Drop the metadata snapshot.

Status
======

<metadata block size> <#used metadata blocks>/<#total metadata blocks>
<current era> <held metadata root | '-'>

========================= ==============================================
metadata block size	  Fixed block size for each metadata block in
			  sectors
#used metadata blocks	  Number of metadata blocks used
#total metadata blocks	  Total number of metadata blocks
current era		  The current era
held metadata root	  The location, in blocks, of the metadata root
			  that has been 'held' for userspace read
			  access. '-' indicates there is no held root
========================= ==============================================

Detailed use case
=================

The scenario of invalidating a cache when rolling back a vendor
snapshot was the primary use case when developing this target:

Taking a vendor snapshot
------------------------

- Send a checkpoint message to the era target
- Make a note of the current era in its status line
- Take vendor snapshot (the era and snapshot should be forever
  associated now).

Rolling back to an vendor snapshot
----------------------------------

- Cache enters passthrough mode (see: dm-cache's docs in cache.txt)
- Rollback vendor storage
- Take metadata snapshot
- Ascertain which blocks have been written since the snapshot was taken
  by checking each block's era
- Invalidate those blocks in the caching software
- Cache returns to writeback/writethrough mode

Memory usage
============

The target uses a bitset to record writes in the current era.  It also
has a spare bitset ready for switching over to a new era.  Other than
that it uses a few 4k blocks for updating metadata::

   (4 * nr_blocks) bytes + buffers

Resilience
==========

Metadata is updated on disk before a write to a previously unwritten
block is performed.  As such dm-era should not be effected by a hard
crash such as power failure.

Userland tools
==============

Userland tools are found in the increasingly poorly named
thin-provisioning-tools project:

    https://github.com/jthornber/thin-provisioning-tools


code:
=============
Device Mapper
=============

.. toctree::
    :maxdepth: 1

    cache-policies
    cache
    delay
    dm-clone
    dm-crypt
    dm-dust
    dm-ebs
    dm-flakey
    dm-ima
    dm-init
    dm-integrity
    dm-io
    dm-log
    dm-queue-length
    dm-raid
    dm-service-time
    dm-uevent
    dm-zoned
    era
    kcopyd
    linear
    log-writes
    persistent-data
    snapshot
    statistics
    striped
    switch
    thin-provisioning
    unstriped
    verity
    writecache
    zero

.. only::  subproject and html

   Indices
   =======

   * :ref:`genindex`


code:
======
kcopyd
======

Kcopyd provides the ability to copy a range of sectors from one block-device
to one or more other block-devices, with an asynchronous completion
notification. It is used by dm-snapshot and dm-mirror.

Users of kcopyd must first create a client and indicate how many memory pages
to set aside for their copy jobs. This is done with a call to
kcopyd_client_create()::

   int kcopyd_client_create(unsigned int num_pages,
                            struct kcopyd_client **result);

To start a copy job, the user must set up io_region structures to describe
the source and destinations of the copy. Each io_region indicates a
block-device along with the starting sector and size of the region. The source
of the copy is given as one io_region structure, and the destinations of the
copy are given as an array of io_region structures::

   struct io_region {
      struct block_device *bdev;
      sector_t sector;
      sector_t count;
   };

To start the copy, the user calls kcopyd_copy(), passing in the client
pointer, pointers to the source and destination io_regions, the name of a
completion callback routine, and a pointer to some context data for the copy::

   int kcopyd_copy(struct kcopyd_client *kc, struct io_region *from,
                   unsigned int num_dests, struct io_region *dests,
                   unsigned int flags, kcopyd_notify_fn fn, void *context);

   typedef void (*kcopyd_notify_fn)(int read_err, unsigned int write_err,
				    void *context);

When the copy completes, kcopyd will call the user's completion routine,
passing back the user's context pointer. It will also indicate if a read or
write error occurred during the copy.

When a user is done with all their copy jobs, they should call
kcopyd_client_destroy() to delete the kcopyd client, which will release the
associated memory pages::

   void kcopyd_client_destroy(struct kcopyd_client *kc);


code:
=========
dm-linear
=========

Device-Mapper's "linear" target maps a linear range of the Device-Mapper
device onto a linear range of another device.  This is the basic building
block of logical volume managers.

Parameters: <dev path> <offset>
    <dev path>:
	Full pathname to the underlying block-device, or a
        "major:minor" device-number.
    <offset>:
	Starting sector within the device.

Example scripts
===============

::

  #!/bin/sh
  # Create an identity mapping for a device
  echo "0 `blockdev --getsz $1` linear $1 0" | dmsetup create identity

::

  #!/bin/sh
  # Join 2 devices together
  size1=`blockdev --getsz $1`
  size2=`blockdev --getsz $2`
  echo "0 $size1 linear $1 0
  $size1 $size2 linear $2 0" | dmsetup create joined

::

  #!/usr/bin/perl -w
  # Split a device into 4M chunks and then join them together in reverse order.

  my $name = "reverse";
  my $extent_size = 4 * 1024 * 2;
  my $dev = $ARGV[0];
  my $table = "";
  my $count = 0;

  if (!defined($dev)) {
          die("Please specify a device.\n");
  }

  my $dev_size = `blockdev --getsz $dev`;
  my $extents = int($dev_size / $extent_size) -
                (($dev_size % $extent_size) ? 1 : 0);

  while ($extents > 0) {
          my $this_start = $count * $extent_size;
          $extents--;
          $count++;
          my $this_offset = $extents * $extent_size;

          $table .= "$this_start $extent_size linear $dev $this_offset\n";
  }

  `echo \"$table\" | dmsetup create $name`;


code:
=============
dm-log-writes
=============

This target takes 2 devices, one to pass all IO to normally, and one to log all
of the write operations to.  This is intended for file system developers wishing
to verify the integrity of metadata or data as the file system is written to.
There is a log_write_entry written for every WRITE request and the target is
able to take arbitrary data from userspace to insert into the log.  The data
that is in the WRITE requests is copied into the log to make the replay happen
exactly as it happened originally.

Log Ordering
============

We log things in order of completion once we are sure the write is no longer in
cache.  This means that normal WRITE requests are not actually logged until the
next REQ_PREFLUSH request.  This is to make it easier for userspace to replay
the log in a way that correlates to what is on disk and not what is in cache,
to make it easier to detect improper waiting/flushing.

This works by attaching all WRITE requests to a list once the write completes.
Once we see a REQ_PREFLUSH request we splice this list onto the request and once
the FLUSH request completes we log all of the WRITEs and then the FLUSH.  Only
completed WRITEs, at the time the REQ_PREFLUSH is issued, are added in order to
simulate the worst case scenario with regard to power failures.  Consider the
following example (W means write, C means complete):

	W1,W2,W3,C3,C2,Wflush,C1,Cflush

The log would show the following:

	W3,W2,flush,W1....

Again this is to simulate what is actually on disk, this allows us to detect
cases where a power failure at a particular point in time would create an
inconsistent file system.

Any REQ_FUA requests bypass this flushing mechanism and are logged as soon as
they complete as those requests will obviously bypass the device cache.

Any REQ_OP_DISCARD requests are treated like WRITE requests.  Otherwise we would
have all the DISCARD requests, and then the WRITE requests and then the FLUSH
request.  Consider the following example:

	WRITE block 1, DISCARD block 1, FLUSH

If we logged DISCARD when it completed, the replay would look like this:

	DISCARD 1, WRITE 1, FLUSH

which isn't quite what happened and wouldn't be caught during the log replay.

Target interface
================

i) Constructor

   log-writes <dev_path> <log_dev_path>

   ============= ==============================================
   dev_path	 Device that all of the IO will go to normally.
   log_dev_path  Device where the log entries are written to.
   ============= ==============================================

ii) Status

    <#logged entries> <highest allocated sector>

    =========================== ========================
    #logged entries	        Number of logged entries
    highest allocated sector    Highest allocated sector
    =========================== ========================

iii) Messages

    mark <description>

	You can use a dmsetup message to set an arbitrary mark in a log.
	For example say you want to fsck a file system after every
	write, but first you need to replay up to the mkfs to make sure
	we're fsck'ing something reasonable, you would do something like
	this::

	  mkfs.btrfs -f /dev/mapper/log
	  dmsetup message log 0 mark mkfs
	  <run test>

	This would allow you to replay the log up to the mkfs mark and
	then replay from that point on doing the fsck check in the
	interval that you want.

	Every log has a mark at the end labeled "dm-log-writes-end".

Userspace component
===================

There is a userspace tool that will replay the log for you in various ways.
It can be found here: https://github.com/josefbacik/log-writes

Example usage
=============

Say you want to test fsync on your file system.  You would do something like
this::

  TABLE="0 $(blockdev --getsz /dev/sdb) log-writes /dev/sdb /dev/sdc"
  dmsetup create log --table "$TABLE"
  mkfs.btrfs -f /dev/mapper/log
  dmsetup message log 0 mark mkfs

  mount /dev/mapper/log /mnt/btrfs-test
  <some test that does fsync at the end>
  dmsetup message log 0 mark fsync
  md5sum /mnt/btrfs-test/foo
  umount /mnt/btrfs-test

  dmsetup remove log
  replay-log --log /dev/sdc --replay /dev/sdb --end-mark fsync
  mount /dev/sdb /mnt/btrfs-test
  md5sum /mnt/btrfs-test/foo
  <verify md5sum's are correct>

  Another option is to do a complicated file system operation and verify the file
  system is consistent during the entire operation.  You could do this with:

  TABLE="0 $(blockdev --getsz /dev/sdb) log-writes /dev/sdb /dev/sdc"
  dmsetup create log --table "$TABLE"
  mkfs.btrfs -f /dev/mapper/log
  dmsetup message log 0 mark mkfs

  mount /dev/mapper/log /mnt/btrfs-test
  <fsstress to dirty the fs>
  btrfs filesystem balance /mnt/btrfs-test
  umount /mnt/btrfs-test
  dmsetup remove log

  replay-log --log /dev/sdc --replay /dev/sdb --end-mark mkfs
  btrfsck /dev/sdb
  replay-log --log /dev/sdc --replay /dev/sdb --start-mark mkfs \
	--fsck "btrfsck /dev/sdb" --check fua

And that will replay the log until it sees a FUA request, run the fsck command
and if the fsck passes it will replay to the next FUA, until it is completed or
the fsck command exists abnormally.


code:
===============
Persistent data
===============

Introduction
============

The more-sophisticated device-mapper targets require complex metadata
that is managed in kernel.  In late 2010 we were seeing that various
different targets were rolling their own data structures, for example:

- Mikulas Patocka's multisnap implementation
- Heinz Mauelshagen's thin provisioning target
- Another btree-based caching target posted to dm-devel
- Another multi-snapshot target based on a design of Daniel Phillips

Maintaining these data structures takes a lot of work, so if possible
we'd like to reduce the number.

The persistent-data library is an attempt to provide a re-usable
framework for people who want to store metadata in device-mapper
targets.  It's currently used by the thin-provisioning target and an
upcoming hierarchical storage target.

Overview
========

The main documentation is in the header files which can all be found
under drivers/md/persistent-data.

The block manager
-----------------

dm-block-manager.[hc]

This provides access to the data on disk in fixed sized-blocks.  There
is a read/write locking interface to prevent concurrent accesses, and
keep data that is being used in the cache.

Clients of persistent-data are unlikely to use this directly.

The transaction manager
-----------------------

dm-transaction-manager.[hc]

This restricts access to blocks and enforces copy-on-write semantics.
The only way you can get hold of a writable block through the
transaction manager is by shadowing an existing block (ie. doing
copy-on-write) or allocating a fresh one.  Shadowing is elided within
the same transaction so performance is reasonable.  The commit method
ensures that all data is flushed before it writes the superblock.
On power failure your metadata will be as it was when last committed.

The Space Maps
--------------

dm-space-map.h
dm-space-map-metadata.[hc]
dm-space-map-disk.[hc]

On-disk data structures that keep track of reference counts of blocks.
Also acts as the allocator of new blocks.  Currently two
implementations: a simpler one for managing blocks on a different
device (eg. thinly-provisioned data blocks); and one for managing
the metadata space.  The latter is complicated by the need to store
its own data within the space it's managing.

The data structures
-------------------

dm-btree.[hc]
dm-btree-remove.c
dm-btree-spine.c
dm-btree-internal.h

Currently there is only one data structure, a hierarchical btree.
There are plans to add more.  For example, something with an
array-like interface would see a lot of use.

The btree is 'hierarchical' in that you can define it to be composed
of nested btrees, and take multiple keys.  For example, the
thin-provisioning target uses a btree with two levels of nesting.
The first maps a device id to a mapping tree, and that in turn maps a
virtual block to a physical block.

Values stored in the btrees can have arbitrary size.  Keys are always
64bits, although nesting allows you to use multiple keys.


code:
==============================
Device-mapper snapshot support
==============================

Device-mapper allows you, without massive data copying:

-  To create snapshots of any block device i.e. mountable, saved states of
   the block device which are also writable without interfering with the
   original content;
-  To create device "forks", i.e. multiple different versions of the
   same data stream.
-  To merge a snapshot of a block device back into the snapshot's origin
   device.

In the first two cases, dm copies only the chunks of data that get
changed and uses a separate copy-on-write (COW) block device for
storage.

For snapshot merge the contents of the COW storage are merged back into
the origin device.

There are three dm targets available:
snapshot, snapshot-origin, and snapshot-merge.

-  snapshot-origin <origin>

which will normally have one or more snapshots based on it.
Reads will be mapped directly to the backing device. For each write, the
original data will be saved in the <COW device> of each snapshot to keep
its visible content unchanged, at least until the <COW device> fills up.

-  snapshot <origin> <COW device> <persistent?> <chunksize>
   [<# feature args> [<arg>]*]

A snapshot of the <origin> block device is created. Changed chunks of
<chunksize> sectors will be stored on the <COW device>.  Writes will
only go to the <COW device>.  Reads will come from the <COW device> or
from <origin> for unchanged data.  <COW device> will often be
smaller than the origin and if it fills up the snapshot will become
useless and be disabled, returning errors.  So it is important to monitor
the amount of free space and expand the <COW device> before it fills up.

<persistent?> is P (Persistent) or N (Not persistent - will not survive
after reboot).  O (Overflow) can be added as a persistent store option
to allow userspace to advertise its support for seeing "Overflow" in the
snapshot status.  So supported store types are "P", "PO" and "N".

The difference between persistent and transient is with transient
snapshots less metadata must be saved on disk - they can be kept in
memory by the kernel.

When loading or unloading the snapshot target, the corresponding
snapshot-origin or snapshot-merge target must be suspended. A failure to
suspend the origin target could result in data corruption.

Optional features:

   discard_zeroes_cow - a discard issued to the snapshot device that
   maps to entire chunks to will zero the corresponding exception(s) in
   the snapshot's exception store.

   discard_passdown_origin - a discard to the snapshot device is passed
   down to the snapshot-origin's underlying device.  This doesn't cause
   copy-out to the snapshot exception store because the snapshot-origin
   target is bypassed.

   The discard_passdown_origin feature depends on the discard_zeroes_cow
   feature being enabled.

-  snapshot-merge <origin> <COW device> <persistent> <chunksize>
   [<# feature args> [<arg>]*]

takes the same table arguments as the snapshot target except it only
works with persistent snapshots.  This target assumes the role of the
"snapshot-origin" target and must not be loaded if the "snapshot-origin"
is still present for <origin>.

Creates a merging snapshot that takes control of the changed chunks
stored in the <COW device> of an existing snapshot, through a handover
procedure, and merges these chunks back into the <origin>.  Once merging
has started (in the background) the <origin> may be opened and the merge
will continue while I/O is flowing to it.  Changes to the <origin> are
deferred until the merging snapshot's corresponding chunk(s) have been
merged.  Once merging has started the snapshot device, associated with
the "snapshot" target, will return -EIO when accessed.

How snapshot is used by LVM2
============================
When you create the first LVM2 snapshot of a volume, four dm devices are used:

1) a device containing the original mapping table of the source volume;
2) a device used as the <COW device>;
3) a "snapshot" device, combining #1 and #2, which is the visible snapshot
   volume;
4) the "original" volume (which uses the device number used by the original
   source volume), whose table is replaced by a "snapshot-origin" mapping
   from device #1.

A fixed naming scheme is used, so with the following commands::

  lvcreate -L 1G -n base volumeGroup
  lvcreate -L 100M --snapshot -n snap volumeGroup/base

we'll have this situation (with volumes in above order)::

  # dmsetup table|grep volumeGroup

  volumeGroup-base-real: 0 2097152 linear 8:19 384
  volumeGroup-snap-cow: 0 204800 linear 8:19 2097536
  volumeGroup-snap: 0 2097152 snapshot 254:11 254:12 P 16
  volumeGroup-base: 0 2097152 snapshot-origin 254:11

  # ls -lL /dev/mapper/volumeGroup-*
  brw-------  1 root root 254, 11 29 ago 18:15 /dev/mapper/volumeGroup-base-real
  brw-------  1 root root 254, 12 29 ago 18:15 /dev/mapper/volumeGroup-snap-cow
  brw-------  1 root root 254, 13 29 ago 18:15 /dev/mapper/volumeGroup-snap
  brw-------  1 root root 254, 10 29 ago 18:14 /dev/mapper/volumeGroup-base

How snapshot-merge is used by LVM2
==================================
A merging snapshot assumes the role of the "snapshot-origin" while
merging.  As such the "snapshot-origin" is replaced with
"snapshot-merge".  The "-real" device is not changed and the "-cow"
device is renamed to <origin name>-cow to aid LVM2's cleanup of the
merging snapshot after it completes.  The "snapshot" that hands over its
COW device to the "snapshot-merge" is deactivated (unless using lvchange
--refresh); but if it is left active it will simply return I/O errors.

A snapshot will merge into its origin with the following command::

  lvconvert --merge volumeGroup/snap

we'll now have this situation::

  # dmsetup table|grep volumeGroup

  volumeGroup-base-real: 0 2097152 linear 8:19 384
  volumeGroup-base-cow: 0 204800 linear 8:19 2097536
  volumeGroup-base: 0 2097152 snapshot-merge 254:11 254:12 P 16

  # ls -lL /dev/mapper/volumeGroup-*
  brw-------  1 root root 254, 11 29 ago 18:15 /dev/mapper/volumeGroup-base-real
  brw-------  1 root root 254, 12 29 ago 18:16 /dev/mapper/volumeGroup-base-cow
  brw-------  1 root root 254, 10 29 ago 18:16 /dev/mapper/volumeGroup-base

How to determine when a merging is complete
===========================================
The snapshot-merge and snapshot status lines end with:

  <sectors_allocated>/<total_sectors> <metadata_sectors>

Both <sectors_allocated> and <total_sectors> include both data and metadata.
During merging, the number of sectors allocated gets smaller and
smaller.  Merging has finished when the number of sectors holding data
is zero, in other words <sectors_allocated> == <metadata_sectors>.

Here is a practical example (using a hybrid of lvm and dmsetup commands)::

  # lvs
    LV      VG          Attr   LSize Origin  Snap%  Move Log Copy%  Convert
    base    volumeGroup owi-a- 4.00g
    snap    volumeGroup swi-a- 1.00g base  18.97

  # dmsetup status volumeGroup-snap
  0 8388608 snapshot 397896/2097152 1560
                                    ^^^^ metadata sectors

  # lvconvert --merge -b volumeGroup/snap
    Merging of volume snap started.

  # lvs volumeGroup/snap
    LV      VG          Attr   LSize Origin  Snap%  Move Log Copy%  Convert
    base    volumeGroup Owi-a- 4.00g          17.23

  # dmsetup status volumeGroup-base
  0 8388608 snapshot-merge 281688/2097152 1104

  # dmsetup status volumeGroup-base
  0 8388608 snapshot-merge 180480/2097152 712

  # dmsetup status volumeGroup-base
  0 8388608 snapshot-merge 16/2097152 16

Merging has finished.

::

  # lvs
    LV      VG          Attr   LSize Origin  Snap%  Move Log Copy%  Convert
    base    volumeGroup owi-a- 4.00g


code:
=============
DM statistics
=============

Device Mapper supports the collection of I/O statistics on user-defined
regions of a DM device.	 If no regions are defined no statistics are
collected so there isn't any performance impact.  Only bio-based DM
devices are currently supported.

Each user-defined region specifies a starting sector, length and step.
Individual statistics will be collected for each step-sized area within
the range specified.

The I/O statistics counters for each step-sized area of a region are
in the same format as `/sys/block/*/stat` or `/proc/diskstats` (see:
Documentation/admin-guide/iostats.rst).  But two extra counters (12 and 13) are
provided: total time spent reading and writing.  When the histogram
argument is used, the 14th parameter is reported that represents the
histogram of latencies.  All these counters may be accessed by sending
the @stats_print message to the appropriate DM device via dmsetup.

The reported times are in milliseconds and the granularity depends on
the kernel ticks.  When the option precise_timestamps is used, the
reported times are in nanoseconds.

Each region has a corresponding unique identifier, which we call a
region_id, that is assigned when the region is created.	 The region_id
must be supplied when querying statistics about the region, deleting the
region, etc.  Unique region_ids enable multiple userspace programs to
request and process statistics for the same DM device without stepping
on each other's data.

The creation of DM statistics will allocate memory via kmalloc or
fallback to using vmalloc space.  At most, 1/4 of the overall system
memory may be allocated by DM statistics.  The admin can see how much
memory is used by reading:

	/sys/module/dm_mod/parameters/stats_current_allocated_bytes

Messages
========

    @stats_create <range> <step> [<number_of_optional_arguments> <optional_arguments>...] [<program_id> [<aux_data>]]
	Create a new region and return the region_id.

	<range>
	  "-"
		whole device
	  "<start_sector>+<length>"
		a range of <length> 512-byte sectors
		starting with <start_sector>.

	<step>
	  "<area_size>"
		the range is subdivided into areas each containing
		<area_size> sectors.
	  "/<number_of_areas>"
		the range is subdivided into the specified
		number of areas.

	<number_of_optional_arguments>
	  The number of optional arguments

	<optional_arguments>
	  The following optional arguments are supported:

	  precise_timestamps
		use precise timer with nanosecond resolution
		instead of the "jiffies" variable.  When this argument is
		used, the resulting times are in nanoseconds instead of
		milliseconds.  Precise timestamps are a little bit slower
		to obtain than jiffies-based timestamps.
	  histogram:n1,n2,n3,n4,...
		collect histogram of latencies.  The
		numbers n1, n2, etc are times that represent the boundaries
		of the histogram.  If precise_timestamps is not used, the
		times are in milliseconds, otherwise they are in
		nanoseconds.  For each range, the kernel will report the
		number of requests that completed within this range. For
		example, if we use "histogram:10,20,30", the kernel will
		report four numbers a:b:c:d. a is the number of requests
		that took 0-10 ms to complete, b is the number of requests
		that took 10-20 ms to complete, c is the number of requests
		that took 20-30 ms to complete and d is the number of
		requests that took more than 30 ms to complete.

	<program_id>
	  An optional parameter.  A name that uniquely identifies
	  the userspace owner of the range.  This groups ranges together
	  so that userspace programs can identify the ranges they
	  created and ignore those created by others.
	  The kernel returns this string back in the output of
	  @stats_list message, but it doesn't use it for anything else.
	  If we omit the number of optional arguments, program id must not
	  be a number, otherwise it would be interpreted as the number of
	  optional arguments.

	<aux_data>
	  An optional parameter.  A word that provides auxiliary data
	  that is useful to the client program that created the range.
	  The kernel returns this string back in the output of
	  @stats_list message, but it doesn't use this value for anything.

    @stats_delete <region_id>
	Delete the region with the specified id.

	<region_id>
	  region_id returned from @stats_create

    @stats_clear <region_id>
	Clear all the counters except the in-flight i/o counters.

	<region_id>
	  region_id returned from @stats_create

    @stats_list [<program_id>]
	List all regions registered with @stats_create.

	<program_id>
	  An optional parameter.
	  If this parameter is specified, only matching regions
	  are returned.
	  If it is not specified, all regions are returned.

	Output format:
	  <region_id>: <start_sector>+<length> <step> <program_id> <aux_data>
	        precise_timestamps histogram:n1,n2,n3,...

	The strings "precise_timestamps" and "histogram" are printed only
	if they were specified when creating the region.

    @stats_print <region_id> [<starting_line> <number_of_lines>]
	Print counters for each step-sized area of a region.

	<region_id>
	  region_id returned from @stats_create

	<starting_line>
	  The index of the starting line in the output.
	  If omitted, all lines are returned.

	<number_of_lines>
	  The number of lines to include in the output.
	  If omitted, all lines are returned.

	Output format for each step-sized area of a region:

	  <start_sector>+<length>
		counters

	  The first 11 counters have the same meaning as
	  `/sys/block/*/stat or /proc/diskstats`.

	  Please refer to Documentation/admin-guide/iostats.rst for details.

	  1. the number of reads completed
	  2. the number of reads merged
	  3. the number of sectors read
	  4. the number of milliseconds spent reading
	  5. the number of writes completed
	  6. the number of writes merged
	  7. the number of sectors written
	  8. the number of milliseconds spent writing
	  9. the number of I/Os currently in progress
	  10. the number of milliseconds spent doing I/Os
	  11. the weighted number of milliseconds spent doing I/Os

	  Additional counters:

	  12. the total time spent reading in milliseconds
	  13. the total time spent writing in milliseconds

    @stats_print_clear <region_id> [<starting_line> <number_of_lines>]
	Atomically print and then clear all the counters except the
	in-flight i/o counters.	 Useful when the client consuming the
	statistics does not want to lose any statistics (those updated
	between printing and clearing).

	<region_id>
	  region_id returned from @stats_create

	<starting_line>
	  The index of the starting line in the output.
	  If omitted, all lines are printed and then cleared.

	<number_of_lines>
	  The number of lines to process.
	  If omitted, all lines are printed and then cleared.

    @stats_set_aux <region_id> <aux_data>
	Store auxiliary data aux_data for the specified region.

	<region_id>
	  region_id returned from @stats_create

	<aux_data>
	  The string that identifies data which is useful to the client
	  program that created the range.  The kernel returns this
	  string back in the output of @stats_list message, but it
	  doesn't use this value for anything.

Examples
========

Subdivide the DM device 'vol' into 100 pieces and start collecting
statistics on them::

  dmsetup message vol 0 @stats_create - /100

Set the auxiliary data string to "foo bar baz" (the escape for each
space must also be escaped, otherwise the shell will consume them)::

  dmsetup message vol 0 @stats_set_aux 0 foo\\ bar\\ baz

List the statistics::

  dmsetup message vol 0 @stats_list

Print the statistics::

  dmsetup message vol 0 @stats_print 0

Delete the statistics::

  dmsetup message vol 0 @stats_delete 0


code:
=========
dm-stripe
=========

Device-Mapper's "striped" target is used to create a striped (i.e. RAID-0)
device across one or more underlying devices. Data is written in "chunks",
with consecutive chunks rotating among the underlying devices. This can
potentially provide improved I/O throughput by utilizing several physical
devices in parallel.

Parameters: <num devs> <chunk size> [<dev path> <offset>]+
    <num devs>:
	Number of underlying devices.
    <chunk size>:
	Size of each chunk of data. Must be at least as
        large as the system's PAGE_SIZE.
    <dev path>:
	Full pathname to the underlying block-device, or a
	"major:minor" device-number.
    <offset>:
	Starting sector within the device.

One or more underlying devices can be specified. The striped device size must
be a multiple of the chunk size multiplied by the number of underlying devices.

Example scripts
===============

::

  #!/usr/bin/perl -w
  # Create a striped device across any number of underlying devices. The device
  # will be called "stripe_dev" and have a chunk-size of 128k.

  my $chunk_size = 128 * 2;
  my $dev_name = "stripe_dev";
  my $num_devs = @ARGV;
  my @devs = @ARGV;
  my ($min_dev_size, $stripe_dev_size, $i);

  if (!$num_devs) {
          die("Specify at least one device\n");
  }

  $min_dev_size = `blockdev --getsz $devs[0]`;
  for ($i = 1; $i < $num_devs; $i++) {
          my $this_size = `blockdev --getsz $devs[$i]`;
          $min_dev_size = ($min_dev_size < $this_size) ?
                          $min_dev_size : $this_size;
  }

  $stripe_dev_size = $min_dev_size * $num_devs;
  $stripe_dev_size -= $stripe_dev_size % ($chunk_size * $num_devs);

  $table = "0 $stripe_dev_size striped $num_devs $chunk_size";
  for ($i = 0; $i < $num_devs; $i++) {
          $table .= " $devs[$i] 0";
  }

  `echo $table | dmsetup create $dev_name`;


code:
=========
dm-switch
=========

The device-mapper switch target creates a device that supports an
arbitrary mapping of fixed-size regions of I/O across a fixed set of
paths.  The path used for any specific region can be switched
dynamically by sending the target a message.

It maps I/O to underlying block devices efficiently when there is a large
number of fixed-sized address regions but there is no simple pattern
that would allow for a compact representation of the mapping such as
dm-stripe.

Background
----------

Dell EqualLogic and some other iSCSI storage arrays use a distributed
frameless architecture.  In this architecture, the storage group
consists of a number of distinct storage arrays ("members") each having
independent controllers, disk storage and network adapters.  When a LUN
is created it is spread across multiple members.  The details of the
spreading are hidden from initiators connected to this storage system.
The storage group exposes a single target discovery portal, no matter
how many members are being used.  When iSCSI sessions are created, each
session is connected to an eth port on a single member.  Data to a LUN
can be sent on any iSCSI session, and if the blocks being accessed are
stored on another member the I/O will be forwarded as required.  This
forwarding is invisible to the initiator.  The storage layout is also
dynamic, and the blocks stored on disk may be moved from member to
member as needed to balance the load.

This architecture simplifies the management and configuration of both
the storage group and initiators.  In a multipathing configuration, it
is possible to set up multiple iSCSI sessions to use multiple network
interfaces on both the host and target to take advantage of the
increased network bandwidth.  An initiator could use a simple round
robin algorithm to send I/O across all paths and let the storage array
members forward it as necessary, but there is a performance advantage to
sending data directly to the correct member.

A device-mapper table already lets you map different regions of a
device onto different targets.  However in this architecture the LUN is
spread with an address region size on the order of 10s of MBs, which
means the resulting table could have more than a million entries and
consume far too much memory.

Using this device-mapper switch target we can now build a two-layer
device hierarchy:

    Upper Tier - Determine which array member the I/O should be sent to.
    Lower Tier - Load balance amongst paths to a particular member.

The lower tier consists of a single dm multipath device for each member.
Each of these multipath devices contains the set of paths directly to
the array member in one priority group, and leverages existing path
selectors to load balance amongst these paths.  We also build a
non-preferred priority group containing paths to other array members for
failover reasons.

The upper tier consists of a single dm-switch device.  This device uses
a bitmap to look up the location of the I/O and choose the appropriate
lower tier device to route the I/O.  By using a bitmap we are able to
use 4 bits for each address range in a 16 member group (which is very
large for us).  This is a much denser representation than the dm table
b-tree can achieve.

Construction Parameters
=======================

    <num_paths> <region_size> <num_optional_args> [<optional_args>...] [<dev_path> <offset>]+
	<num_paths>
	    The number of paths across which to distribute the I/O.

	<region_size>
	    The number of 512-byte sectors in a region. Each region can be redirected
	    to any of the available paths.

	<num_optional_args>
	    The number of optional arguments. Currently, no optional arguments
	    are supported and so this must be zero.

	<dev_path>
	    The block device that represents a specific path to the device.

	<offset>
	    The offset of the start of data on the specific <dev_path> (in units
	    of 512-byte sectors). This number is added to the sector number when
	    forwarding the request to the specific path. Typically it is zero.

Messages
========

set_region_mappings <index>:<path_nr> [<index>]:<path_nr> [<index>]:<path_nr>...

Modify the region table by specifying which regions are redirected to
which paths.

<index>
    The region number (region size was specified in constructor parameters).
    If index is omitted, the next region (previous index + 1) is used.
    Expressed in hexadecimal (WITHOUT any prefix like 0x).

<path_nr>
    The path number in the range 0 ... (<num_paths> - 1).
    Expressed in hexadecimal (WITHOUT any prefix like 0x).

R<n>,<m>
    This parameter allows repetitive patterns to be loaded quickly. <n> and <m>
    are hexadecimal numbers. The last <n> mappings are repeated in the next <m>
    slots.

Status
======

No status line is reported.

Example
=======

Assume that you have volumes vg1/switch0 vg1/switch1 vg1/switch2 with
the same size.

Create a switch device with 64kB region size::

    dmsetup create switch --table "0 `blockdev --getsz /dev/vg1/switch0`
	switch 3 128 0 /dev/vg1/switch0 0 /dev/vg1/switch1 0 /dev/vg1/switch2 0"

Set mappings for the first 7 entries to point to devices switch0, switch1,
switch2, switch0, switch1, switch2, switch1::

    dmsetup message switch 0 set_region_mappings 0:0 :1 :2 :0 :1 :2 :1

Set repetitive mapping. This command::

    dmsetup message switch 0 set_region_mappings 1000:1 :2 R2,10

is equivalent to::

    dmsetup message switch 0 set_region_mappings 1000:1 :2 :1 :2 :1 :2 :1 :2 \
	:1 :2 :1 :2 :1 :2 :1 :2 :1 :2


code:
=================
Thin provisioning
=================

Introduction
============

This document describes a collection of device-mapper targets that
between them implement thin-provisioning and snapshots.

The main highlight of this implementation, compared to the previous
implementation of snapshots, is that it allows many virtual devices to
be stored on the same data volume.  This simplifies administration and
allows the sharing of data between volumes, thus reducing disk usage.

Another significant feature is support for an arbitrary depth of
recursive snapshots (snapshots of snapshots of snapshots ...).  The
previous implementation of snapshots did this by chaining together
lookup tables, and so performance was O(depth).  This new
implementation uses a single data structure to avoid this degradation
with depth.  Fragmentation may still be an issue, however, in some
scenarios.

Metadata is stored on a separate device from data, giving the
administrator some freedom, for example to:

- Improve metadata resilience by storing metadata on a mirrored volume
  but data on a non-mirrored one.

- Improve performance by storing the metadata on SSD.

Status
======

These targets are considered safe for production use.  But different use
cases will have different performance characteristics, for example due
to fragmentation of the data volume.

If you find this software is not performing as expected please mail
dm-devel@redhat.com with details and we'll try our best to improve
things for you.

Userspace tools for checking and repairing the metadata have been fully
developed and are available as 'thin_check' and 'thin_repair'.  The name
of the package that provides these utilities varies by distribution (on
a Red Hat distribution it is named 'device-mapper-persistent-data').

Cookbook
========

This section describes some quick recipes for using thin provisioning.
They use the dmsetup program to control the device-mapper driver
directly.  End users will be advised to use a higher-level volume
manager such as LVM2 once support has been added.

Pool device
-----------

The pool device ties together the metadata volume and the data volume.
It maps I/O linearly to the data volume and updates the metadata via
two mechanisms:

- Function calls from the thin targets

- Device-mapper 'messages' from userspace which control the creation of new
  virtual devices amongst other things.

Setting up a fresh pool device
------------------------------

Setting up a pool device requires a valid metadata device, and a
data device.  If you do not have an existing metadata device you can
make one by zeroing the first 4k to indicate empty metadata.

    dd if=/dev/zero of=$metadata_dev bs=4096 count=1

The amount of metadata you need will vary according to how many blocks
are shared between thin devices (i.e. through snapshots).  If you have
less sharing than average you'll need a larger-than-average metadata device.

As a guide, we suggest you calculate the number of bytes to use in the
metadata device as 48 * $data_dev_size / $data_block_size but round it up
to 2MB if the answer is smaller.  If you're creating large numbers of
snapshots which are recording large amounts of change, you may find you
need to increase this.

The largest size supported is 16GB: If the device is larger,
a warning will be issued and the excess space will not be used.

Reloading a pool table
----------------------

You may reload a pool's table, indeed this is how the pool is resized
if it runs out of space.  (N.B. While specifying a different metadata
device when reloading is not forbidden at the moment, things will go
wrong if it does not route I/O to exactly the same on-disk location as
previously.)

Using an existing pool device
-----------------------------

::

    dmsetup create pool \
	--table "0 20971520 thin-pool $metadata_dev $data_dev \
		 $data_block_size $low_water_mark"

$data_block_size gives the smallest unit of disk space that can be
allocated at a time expressed in units of 512-byte sectors.
$data_block_size must be between 128 (64KB) and 2097152 (1GB) and a
multiple of 128 (64KB).  $data_block_size cannot be changed after the
thin-pool is created.  People primarily interested in thin provisioning
may want to use a value such as 1024 (512KB).  People doing lots of
snapshotting may want a smaller value such as 128 (64KB).  If you are
not zeroing newly-allocated data, a larger $data_block_size in the
region of 256000 (128MB) is suggested.

$low_water_mark is expressed in blocks of size $data_block_size.  If
free space on the data device drops below this level then a dm event
will be triggered which a userspace daemon should catch allowing it to
extend the pool device.  Only one such event will be sent.

No special event is triggered if a just resumed device's free space is below
the low water mark. However, resuming a device always triggers an
event; a userspace daemon should verify that free space exceeds the low
water mark when handling this event.

A low water mark for the metadata device is maintained in the kernel and
will trigger a dm event if free space on the metadata device drops below
it.

Updating on-disk metadata
-------------------------

On-disk metadata is committed every time a FLUSH or FUA bio is written.
If no such requests are made then commits will occur every second.  This
means the thin-provisioning target behaves like a physical disk that has
a volatile write cache.  If power is lost you may lose some recent
writes.  The metadata should always be consistent in spite of any crash.

If data space is exhausted the pool will either error or queue IO
according to the configuration (see: error_if_no_space).  If metadata
space is exhausted or a metadata operation fails: the pool will error IO
until the pool is taken offline and repair is performed to 1) fix any
potential inconsistencies and 2) clear the flag that imposes repair.
Once the pool's metadata device is repaired it may be resized, which
will allow the pool to return to normal operation.  Note that if a pool
is flagged as needing repair, the pool's data and metadata devices
cannot be resized until repair is performed.  It should also be noted
that when the pool's metadata space is exhausted the current metadata
transaction is aborted.  Given that the pool will cache IO whose
completion may have already been acknowledged to upper IO layers
(e.g. filesystem) it is strongly suggested that consistency checks
(e.g. fsck) be performed on those layers when repair of the pool is
required.

Thin provisioning
-----------------

i) Creating a new thinly-provisioned volume.

  To create a new thinly- provisioned volume you must send a message to an
  active pool device, /dev/mapper/pool in this example::

    dmsetup message /dev/mapper/pool 0 "create_thin 0"

  Here '0' is an identifier for the volume, a 24-bit number.  It's up
  to the caller to allocate and manage these identifiers.  If the
  identifier is already in use, the message will fail with -EEXIST.

ii) Using a thinly-provisioned volume.

  Thinly-provisioned volumes are activated using the 'thin' target::

    dmsetup create thin --table "0 2097152 thin /dev/mapper/pool 0"

  The last parameter is the identifier for the thinp device.

Internal snapshots
------------------

i) Creating an internal snapshot.

  Snapshots are created with another message to the pool.

  N.B.  If the origin device that you wish to snapshot is active, you
  must suspend it before creating the snapshot to avoid corruption.
  This is NOT enforced at the moment, so please be careful!

  ::

    dmsetup suspend /dev/mapper/thin
    dmsetup message /dev/mapper/pool 0 "create_snap 1 0"
    dmsetup resume /dev/mapper/thin

  Here '1' is the identifier for the volume, a 24-bit number.  '0' is the
  identifier for the origin device.

ii) Using an internal snapshot.

  Once created, the user doesn't have to worry about any connection
  between the origin and the snapshot.  Indeed the snapshot is no
  different from any other thinly-provisioned device and can be
  snapshotted itself via the same method.  It's perfectly legal to
  have only one of them active, and there's no ordering requirement on
  activating or removing them both.  (This differs from conventional
  device-mapper snapshots.)

  Activate it exactly the same way as any other thinly-provisioned volume::

    dmsetup create snap --table "0 2097152 thin /dev/mapper/pool 1"

External snapshots
------------------

You can use an external **read only** device as an origin for a
thinly-provisioned volume.  Any read to an unprovisioned area of the
thin device will be passed through to the origin.  Writes trigger
the allocation of new blocks as usual.

One use case for this is VM hosts that want to run guests on
thinly-provisioned volumes but have the base image on another device
(possibly shared between many VMs).

You must not write to the origin device if you use this technique!
Of course, you may write to the thin device and take internal snapshots
of the thin volume.

i) Creating a snapshot of an external device

  This is the same as creating a thin device.
  You don't mention the origin at this stage.

  ::

    dmsetup message /dev/mapper/pool 0 "create_thin 0"

ii) Using a snapshot of an external device.

  Append an extra parameter to the thin target specifying the origin::

    dmsetup create snap --table "0 2097152 thin /dev/mapper/pool 0 /dev/image"

  N.B. All descendants (internal snapshots) of this snapshot require the
  same extra origin parameter.

Deactivation
------------

All devices using a pool must be deactivated before the pool itself
can be.

::

    dmsetup remove thin
    dmsetup remove snap
    dmsetup remove pool

Reference
=========

'thin-pool' target
------------------

i) Constructor

    ::

      thin-pool <metadata dev> <data dev> <data block size (sectors)> \
	        <low water mark (blocks)> [<number of feature args> [<arg>]*]

    Optional feature arguments:

      skip_block_zeroing:
	Skip the zeroing of newly-provisioned blocks.

      ignore_discard:
	Disable discard support.

      no_discard_passdown:
	Don't pass discards down to the underlying
	data device, but just remove the mapping.

      read_only:
		 Don't allow any changes to be made to the pool
		 metadata.  This mode is only available after the
		 thin-pool has been created and first used in full
		 read/write mode.  It cannot be specified on initial
		 thin-pool creation.

      error_if_no_space:
	Error IOs, instead of queueing, if no space.

    Data block size must be between 64KB (128 sectors) and 1GB
    (2097152 sectors) inclusive.

ii) Status

    ::

      <transaction id> <used metadata blocks>/<total metadata blocks>
      <used data blocks>/<total data blocks> <held metadata root>
      ro|rw|out_of_data_space [no_]discard_passdown [error|queue]_if_no_space
      needs_check|- metadata_low_watermark

    transaction id:
	A 64-bit number used by userspace to help synchronise with metadata
	from volume managers.

    used data blocks / total data blocks
	If the number of free blocks drops below the pool's low water mark a
	dm event will be sent to userspace.  This event is edge-triggered and
	it will occur only once after each resume so volume manager writers
	should register for the event and then check the target's status.

    held metadata root:
	The location, in blocks, of the metadata root that has been
	'held' for userspace read access.  '-' indicates there is no
	held root.

    discard_passdown|no_discard_passdown
	Whether or not discards are actually being passed down to the
	underlying device.  When this is enabled when loading the table,
	it can get disabled if the underlying device doesn't support it.

    ro|rw|out_of_data_space
	If the pool encounters certain types of device failures it will
	drop into a read-only metadata mode in which no changes to
	the pool metadata (like allocating new blocks) are permitted.

	In serious cases where even a read-only mode is deemed unsafe
	no further I/O will be permitted and the status will just
	contain the string 'Fail'.  The userspace recovery tools
	should then be used.

    error_if_no_space|queue_if_no_space
	If the pool runs out of data or metadata space, the pool will
	either queue or error the IO destined to the data device.  The
	default is to queue the IO until more space is added or the
	'no_space_timeout' expires.  The 'no_space_timeout' dm-thin-pool
	module parameter can be used to change this timeout -- it
	defaults to 60 seconds but may be disabled using a value of 0.

    needs_check
	A metadata operation has failed, resulting in the needs_check
	flag being set in the metadata's superblock.  The metadata
	device must be deactivated and checked/repaired before the
	thin-pool can be made fully operational again.  '-' indicates
	needs_check is not set.

    metadata_low_watermark:
	Value of metadata low watermark in blocks.  The kernel sets this
	value internally but userspace needs to know this value to
	determine if an event was caused by crossing this threshold.

iii) Messages

    create_thin <dev id>
	Create a new thinly-provisioned device.
	<dev id> is an arbitrary unique 24-bit identifier chosen by
	the caller.

    create_snap <dev id> <origin id>
	Create a new snapshot of another thinly-provisioned device.
	<dev id> is an arbitrary unique 24-bit identifier chosen by
	the caller.
	<origin id> is the identifier of the thinly-provisioned device
	of which the new device will be a snapshot.

    delete <dev id>
	Deletes a thin device.  Irreversible.

    set_transaction_id <current id> <new id>
	Userland volume managers, such as LVM, need a way to
	synchronise their external metadata with the internal metadata of the
	pool target.  The thin-pool target offers to store an
	arbitrary 64-bit transaction id and return it on the target's
	status line.  To avoid races you must provide what you think
	the current transaction id is when you change it with this
	compare-and-swap message.

    reserve_metadata_snap
        Reserve a copy of the data mapping btree for use by userland.
        This allows userland to inspect the mappings as they were when
        this message was executed.  Use the pool's status command to
        get the root block associated with the metadata snapshot.

    release_metadata_snap
        Release a previously reserved copy of the data mapping btree.

'thin' target
-------------

i) Constructor

    ::

        thin <pool dev> <dev id> [<external origin dev>]

    pool dev:
	the thin-pool device, e.g. /dev/mapper/my_pool or 253:0

    dev id:
	the internal device identifier of the device to be
	activated.

    external origin dev:
	an optional block device outside the pool to be treated as a
	read-only snapshot origin: reads to unprovisioned areas of the
	thin target will be mapped to this device.

The pool doesn't store any size against the thin devices.  If you
load a thin target that is smaller than you've been using previously,
then you'll have no access to blocks mapped beyond the end.  If you
load a target that is bigger than before, then extra blocks will be
provisioned as and when needed.

ii) Status

    <nr mapped sectors> <highest mapped sector>
	If the pool has encountered device errors and failed, the status
	will just contain the string 'Fail'.  The userspace recovery
	tools should then be used.

    In the case where <nr mapped sectors> is 0, there is no highest
    mapped sector and the value of <highest mapped sector> is unspecified.


code:
================================
Device-mapper "unstriped" target
================================

Introduction
============

The device-mapper "unstriped" target provides a transparent mechanism to
unstripe a device-mapper "striped" target to access the underlying disks
without having to touch the true backing block-device.  It can also be
used to unstripe a hardware RAID-0 to access backing disks.

Parameters:
<number of stripes> <chunk size> <stripe #> <dev_path> <offset>

<number of stripes>
        The number of stripes in the RAID 0.

<chunk size>
	The amount of 512B sectors in the chunk striping.

<dev_path>
	The block device you wish to unstripe.

<stripe #>
        The stripe number within the device that corresponds to physical
        drive you wish to unstripe.  This must be 0 indexed.

Why use this module?
====================

An example of undoing an existing dm-stripe
-------------------------------------------

This small bash script will setup 4 loop devices and use the existing
striped target to combine the 4 devices into one.  It then will use
the unstriped target ontop of the striped device to access the
individual backing loop devices.  We write data to the newly exposed
unstriped devices and verify the data written matches the correct
underlying device on the striped array::

  #!/bin/bash

  MEMBER_SIZE=$((128 * 1024 * 1024))
  NUM=4
  SEQ_END=$((${NUM}-1))
  CHUNK=256
  BS=4096

  RAID_SIZE=$((${MEMBER_SIZE}*${NUM}/512))
  DM_PARMS="0 ${RAID_SIZE} striped ${NUM} ${CHUNK}"
  COUNT=$((${MEMBER_SIZE} / ${BS}))

  for i in $(seq 0 ${SEQ_END}); do
    dd if=/dev/zero of=member-${i} bs=${MEMBER_SIZE} count=1 oflag=direct
    losetup /dev/loop${i} member-${i}
    DM_PARMS+=" /dev/loop${i} 0"
  done

  echo $DM_PARMS | dmsetup create raid0
  for i in $(seq 0 ${SEQ_END}); do
    echo "0 1 unstriped ${NUM} ${CHUNK} ${i} /dev/mapper/raid0 0" | dmsetup create set-${i}
  done;

  for i in $(seq 0 ${SEQ_END}); do
    dd if=/dev/urandom of=/dev/mapper/set-${i} bs=${BS} count=${COUNT} oflag=direct
    diff /dev/mapper/set-${i} member-${i}
  done;

  for i in $(seq 0 ${SEQ_END}); do
    dmsetup remove set-${i}
  done

  dmsetup remove raid0

  for i in $(seq 0 ${SEQ_END}); do
    losetup -d /dev/loop${i}
    rm -f member-${i}
  done

Another example
---------------

Intel NVMe drives contain two cores on the physical device.
Each core of the drive has segregated access to its LBA range.
The current LBA model has a RAID 0 128k chunk on each core, resulting
in a 256k stripe across the two cores::

   Core 0:       Core 1:
  __________    __________
  | LBA 512|    | LBA 768|
  | LBA 0  |    | LBA 256|
  ----------    ----------

The purpose of this unstriping is to provide better QoS in noisy
neighbor environments. When two partitions are created on the
aggregate drive without this unstriping, reads on one partition
can affect writes on another partition.  This is because the partitions
are striped across the two cores.  When we unstripe this hardware RAID 0
and make partitions on each new exposed device the two partitions are now
physically separated.

With the dm-unstriped target we're able to segregate an fio script that
has read and write jobs that are independent of each other.  Compared to
when we run the test on a combined drive with partitions, we were able
to get a 92% reduction in read latency using this device mapper target.

Example dmsetup usage
=====================

unstriped ontop of Intel NVMe device that has 2 cores
-----------------------------------------------------

::

  dmsetup create nvmset0 --table '0 512 unstriped 2 256 0 /dev/nvme0n1 0'
  dmsetup create nvmset1 --table '0 512 unstriped 2 256 1 /dev/nvme0n1 0'

There will now be two devices that expose Intel NVMe core 0 and 1
respectively::

  /dev/mapper/nvmset0
  /dev/mapper/nvmset1

unstriped ontop of striped with 4 drives using 128K chunk size
--------------------------------------------------------------

::

  dmsetup create raid_disk0 --table '0 512 unstriped 4 256 0 /dev/mapper/striped 0'
  dmsetup create raid_disk1 --table '0 512 unstriped 4 256 1 /dev/mapper/striped 0'
  dmsetup create raid_disk2 --table '0 512 unstriped 4 256 2 /dev/mapper/striped 0'
  dmsetup create raid_disk3 --table '0 512 unstriped 4 256 3 /dev/mapper/striped 0'


code:
=========
dm-verity
=========

Device-Mapper's "verity" target provides transparent integrity checking of
block devices using a cryptographic digest provided by the kernel crypto API.
This target is read-only.

Construction Parameters
=======================

::

    <version> <dev> <hash_dev>
    <data_block_size> <hash_block_size>
    <num_data_blocks> <hash_start_block>
    <algorithm> <digest> <salt>
    [<#opt_params> <opt_params>]

<version>
    This is the type of the on-disk hash format.

    0 is the original format used in the Chromium OS.
      The salt is appended when hashing, digests are stored continuously and
      the rest of the block is padded with zeroes.

    1 is the current format that should be used for new devices.
      The salt is prepended when hashing and each digest is
      padded with zeroes to the power of two.

<dev>
    This is the device containing data, the integrity of which needs to be
    checked.  It may be specified as a path, like /dev/sdaX, or a device number,
    <major>:<minor>.

<hash_dev>
    This is the device that supplies the hash tree data.  It may be
    specified similarly to the device path and may be the same device.  If the
    same device is used, the hash_start should be outside the configured
    dm-verity device.

<data_block_size>
    The block size on a data device in bytes.
    Each block corresponds to one digest on the hash device.

<hash_block_size>
    The size of a hash block in bytes.

<num_data_blocks>
    The number of data blocks on the data device.  Additional blocks are
    inaccessible.  You can place hashes to the same partition as data, in this
    case hashes are placed after <num_data_blocks>.

<hash_start_block>
    This is the offset, in <hash_block_size>-blocks, from the start of hash_dev
    to the root block of the hash tree.

<algorithm>
    The cryptographic hash algorithm used for this device.  This should
    be the name of the algorithm, like "sha1".

<digest>
    The hexadecimal encoding of the cryptographic hash of the root hash block
    and the salt.  This hash should be trusted as there is no other authenticity
    beyond this point.

<salt>
    The hexadecimal encoding of the salt value.

<#opt_params>
    Number of optional parameters. If there are no optional parameters,
    the optional parameters section can be skipped or #opt_params can be zero.
    Otherwise #opt_params is the number of following arguments.

    Example of optional parameters section:
        1 ignore_corruption

ignore_corruption
    Log corrupted blocks, but allow read operations to proceed normally.

restart_on_corruption
    Restart the system when a corrupted block is discovered. This option is
    not compatible with ignore_corruption and requires user space support to
    avoid restart loops.

panic_on_corruption
    Panic the device when a corrupted block is discovered. This option is
    not compatible with ignore_corruption and restart_on_corruption.

ignore_zero_blocks
    Do not verify blocks that are expected to contain zeroes and always return
    zeroes instead. This may be useful if the partition contains unused blocks
    that are not guaranteed to contain zeroes.

use_fec_from_device <fec_dev>
    Use forward error correction (FEC) to recover from corruption if hash
    verification fails. Use encoding data from the specified device. This
    may be the same device where data and hash blocks reside, in which case
    fec_start must be outside data and hash areas.

    If the encoding data covers additional metadata, it must be accessible
    on the hash device after the hash blocks.

    Note: block sizes for data and hash devices must match. Also, if the
    verity <dev> is encrypted the <fec_dev> should be too.

fec_roots <num>
    Number of generator roots. This equals to the number of parity bytes in
    the encoding data. For example, in RS(M, N) encoding, the number of roots
    is M-N.

fec_blocks <num>
    The number of encoding data blocks on the FEC device. The block size for
    the FEC device is <data_block_size>.

fec_start <offset>
    This is the offset, in <data_block_size> blocks, from the start of the
    FEC device to the beginning of the encoding data.

check_at_most_once
    Verify data blocks only the first time they are read from the data device,
    rather than every time.  This reduces the overhead of dm-verity so that it
    can be used on systems that are memory and/or CPU constrained.  However, it
    provides a reduced level of security because only offline tampering of the
    data device's content will be detected, not online tampering.

    Hash blocks are still verified each time they are read from the hash device,
    since verification of hash blocks is less performance critical than data
    blocks, and a hash block will not be verified any more after all the data
    blocks it covers have been verified anyway.

root_hash_sig_key_desc <key_description>
    This is the description of the USER_KEY that the kernel will lookup to get
    the pkcs7 signature of the roothash. The pkcs7 signature is used to validate
    the root hash during the creation of the device mapper block device.
    Verification of roothash depends on the config DM_VERITY_VERIFY_ROOTHASH_SIG
    being set in the kernel.  The signatures are checked against the builtin
    trusted keyring by default, or the secondary trusted keyring if
    DM_VERITY_VERIFY_ROOTHASH_SIG_SECONDARY_KEYRING is set.  The secondary
    trusted keyring includes by default the builtin trusted keyring, and it can
    also gain new certificates at run time if they are signed by a certificate
    already in the secondary trusted keyring.

try_verify_in_tasklet
    If verity hashes are in cache, verify data blocks in kernel tasklet instead
    of workqueue. This option can reduce IO latency.

Theory of operation
===================

dm-verity is meant to be set up as part of a verified boot path.  This
may be anything ranging from a boot using tboot or trustedgrub to just
booting from a known-good device (like a USB drive or CD).

When a dm-verity device is configured, it is expected that the caller
has been authenticated in some way (cryptographic signatures, etc).
After instantiation, all hashes will be verified on-demand during
disk access.  If they cannot be verified up to the root node of the
tree, the root hash, then the I/O will fail.  This should detect
tampering with any data on the device and the hash data.

Cryptographic hashes are used to assert the integrity of the device on a
per-block basis. This allows for a lightweight hash computation on first read
into the page cache. Block hashes are stored linearly, aligned to the nearest
block size.

If forward error correction (FEC) support is enabled any recovery of
corrupted data will be verified using the cryptographic hash of the
corresponding data. This is why combining error correction with
integrity checking is essential.

Hash Tree
---------

Each node in the tree is a cryptographic hash.  If it is a leaf node, the hash
of some data block on disk is calculated. If it is an intermediary node,
the hash of a number of child nodes is calculated.

Each entry in the tree is a collection of neighboring nodes that fit in one
block.  The number is determined based on block_size and the size of the
selected cryptographic digest algorithm.  The hashes are linearly-ordered in
this entry and any unaligned trailing space is ignored but included when
calculating the parent node.

The tree looks something like:

	alg = sha256, num_blocks = 32768, block_size = 4096

::

                                 [   root    ]
                                /    . . .    \
                     [entry_0]                 [entry_1]
                    /  . . .  \                 . . .   \
         [entry_0_0]   . . .  [entry_0_127]    . . . .  [entry_1_127]
           / ... \             /   . . .  \             /           \
     blk_0 ... blk_127  blk_16256   blk_16383      blk_32640 . . . blk_32767

On-disk format
==============

The verity kernel code does not read the verity metadata on-disk header.
It only reads the hash blocks which directly follow the header.
It is expected that a user-space tool will verify the integrity of the
verity header.

Alternatively, the header can be omitted and the dmsetup parameters can
be passed via the kernel command-line in a rooted chain of trust where
the command-line is verified.

Directly following the header (and with sector number padded to the next hash
block boundary) are the hash blocks which are stored a depth at a time
(starting from the root), sorted in order of increasing index.

The full specification of kernel parameters and on-disk metadata format
is available at the cryptsetup project's wiki page

  https://gitlab.com/cryptsetup/cryptsetup/wikis/DMVerity

Status
======
V (for Valid) is returned if every check performed so far was valid.
If any check failed, C (for Corruption) is returned.

Example
=======
Set up a device::

  # dmsetup create vroot --readonly --table \
    "0 2097152 verity 1 /dev/sda1 /dev/sda2 4096 4096 262144 1 sha256 "\
    "4392712ba01368efdf14b05c76f9e4df0d53664630b5d48632ed17a137f39076 "\
    "1234000000000000000000000000000000000000000000000000000000000000"

A command line tool veritysetup is available to compute or verify
the hash tree or activate the kernel device. This is available from
the cryptsetup upstream repository https://gitlab.com/cryptsetup/cryptsetup/
(as a libcryptsetup extension).

Create hash on the device::

  # veritysetup format /dev/sda1 /dev/sda2
  ...
  Root hash: 4392712ba01368efdf14b05c76f9e4df0d53664630b5d48632ed17a137f39076

Activate the device::

  # veritysetup create vroot /dev/sda1 /dev/sda2 \
    4392712ba01368efdf14b05c76f9e4df0d53664630b5d48632ed17a137f39076


code:
=================
Writecache target
=================

The writecache target caches writes on persistent memory or on SSD. It
doesn't cache reads because reads are supposed to be cached in page cache
in normal RAM.

When the device is constructed, the first sector should be zeroed or the
first sector should contain valid superblock from previous invocation.

Constructor parameters:

1. type of the cache device - "p" or "s"
	- p - persistent memory
	- s - SSD
2. the underlying device that will be cached
3. the cache device
4. block size (4096 is recommended; the maximum block size is the page
   size)
5. the number of optional parameters (the parameters with an argument
   count as two)

	start_sector n		(default: 0)
		offset from the start of cache device in 512-byte sectors
	high_watermark n	(default: 50)
		start writeback when the number of used blocks reach this
		watermark
	low_watermark x		(default: 45)
		stop writeback when the number of used blocks drops below
		this watermark
	writeback_jobs n	(default: unlimited)
		limit the number of blocks that are in flight during
		writeback. Setting this value reduces writeback
		throughput, but it may improve latency of read requests
	autocommit_blocks n	(default: 64 for pmem, 65536 for ssd)
		when the application writes this amount of blocks without
		issuing the FLUSH request, the blocks are automatically
		committed
	autocommit_time ms	(default: 1000)
		autocommit time in milliseconds. The data is automatically
		committed if this time passes and no FLUSH request is
		received
	fua			(by default on)
		applicable only to persistent memory - use the FUA flag
		when writing data from persistent memory back to the
		underlying device
	nofua
		applicable only to persistent memory - don't use the FUA
		flag when writing back data and send the FLUSH request
		afterwards

		- some underlying devices perform better with fua, some
		  with nofua. The user should test it
	cleaner
		when this option is activated (either in the constructor
		arguments or by a message), the cache will not promote
		new writes (however, writes to already cached blocks are
		promoted, to avoid data corruption due to misordered
		writes) and it will gradually writeback any cached
		data. The userspace can then monitor the cleaning
		process with "dmsetup status". When the number of cached
		blocks drops to zero, userspace can unload the
		dm-writecache target and replace it with dm-linear or
		other targets.
	max_age n
		specifies the maximum age of a block in milliseconds. If
		a block is stored in the cache for too long, it will be
		written to the underlying device and cleaned up.
	metadata_only
		only metadata is promoted to the cache. This option
		improves performance for heavier REQ_META workloads.
	pause_writeback n	(default: 3000)
		pause writeback if there was some write I/O redirected to
		the origin volume in the last n milliseconds

Status:

1. error indicator - 0 if there was no error, otherwise error number
2. the number of blocks
3. the number of free blocks
4. the number of blocks under writeback
5. the number of read blocks
6. the number of read blocks that hit the cache
7. the number of write blocks
8. the number of write blocks that hit uncommitted block
9. the number of write blocks that hit committed block
10. the number of write blocks that bypass the cache
11. the number of write blocks that are allocated in the cache
12. the number of write requests that are blocked on the freelist
13. the number of flush requests
14. the number of discarded blocks

Messages:
	flush
		Flush the cache device. The message returns successfully
		if the cache device was flushed without an error
	flush_on_suspend
		Flush the cache device on next suspend. Use this message
		when you are going to remove the cache device. The proper
		sequence for removing the cache device is:

		1. send the "flush_on_suspend" message
		2. load an inactive table with a linear target that maps
		   to the underlying device
		3. suspend the device
		4. ask for status and verify that there are no errors
		5. resume the device, so that it will use the linear
		   target
		6. the cache device is now inactive and it can be deleted
	cleaner
		See above "cleaner" constructor documentation.
	clear_stats
		Clear the statistics that are reported on the status line


code:
=======
dm-zero
=======

Device-Mapper's "zero" target provides a block-device that always returns
zero'd data on reads and silently drops writes. This is similar behavior to
/dev/zero, but as a block-device instead of a character-device.

Dm-zero has no target-specific parameters.

One very interesting use of dm-zero is for creating "sparse" devices in
conjunction with dm-snapshot. A sparse device reports a device-size larger
than the amount of actual storage space available for that device. A user can
write data anywhere within the sparse device and read it back like a normal
device. Reads to previously unwritten areas will return a zero'd buffer. When
enough data has been written to fill up the actual storage space, the sparse
device is deactivated. This can be very useful for testing device and
filesystem limitations.

To create a sparse device, start by creating a dm-zero device that's the
desired size of the sparse device. For this example, we'll assume a 10TB
sparse device::

  TEN_TERABYTES=`expr 10 \* 1024 \* 1024 \* 1024 \* 2`   # 10 TB in sectors
  echo "0 $TEN_TERABYTES zero" | dmsetup create zero1

Then create a snapshot of the zero device, using any available block-device as
the COW device. The size of the COW device will determine the amount of real
space available to the sparse device. For this example, we'll assume /dev/sdb1
is an available 10GB partition::

  echo "0 $TEN_TERABYTES snapshot /dev/mapper/zero1 /dev/sdb1 p 128" | \
     dmsetup create sparse1

This will create a 10TB sparse device called /dev/mapper/sparse1 that has
10GB of actual storage space available. If more than 10GB of data is written
to this device, it will start returning I/O errors.


code:
.. _admin_devices:

Linux allocated devices (4.x+ version)
======================================

This list is the Linux Device List, the official registry of allocated
device numbers and ``/dev`` directory nodes for the Linux operating
system.

The version of this document at lanana.org is no longer maintained.  This
version in the mainline Linux kernel is the master document.  Updates
shall be sent as patches to the kernel maintainers (see the
:ref:`Documentation/process/submitting-patches.rst <submittingpatches>` document).
Specifically explore the sections titled "CHAR and MISC DRIVERS", and
"BLOCK LAYER" in the MAINTAINERS file to find the right maintainers
to involve for character and block devices.

This document is included by reference into the Filesystem Hierarchy
Standard (FHS).	 The FHS is available from https://www.pathname.com/fhs/.

Allocations marked (68k/Amiga) apply to Linux/68k on the Amiga
platform only.	Allocations marked (68k/Atari) apply to Linux/68k on
the Atari platform only.

This document is in the public domain.	The authors requests, however,
that semantically altered versions are not distributed without
permission of the authors, assuming the authors can be contacted without
an unreasonable effort.

.. attention::

  DEVICE DRIVERS AUTHORS PLEASE READ THIS

  Linux now has extensive support for dynamic allocation of device numbering
  and can use ``sysfs`` and ``udev`` (``systemd``) to handle the naming needs.
  There are still some exceptions in the serial and boot device area. Before
  asking   for a device number make sure you actually need one.

  To have a major number allocated, or a minor number in situations
  where that applies (e.g. busmice), please submit a patch and send to
  the authors as indicated above.

  Keep the description of the device *in the same format
  as this list*. The reason for this is that it is the only way we have
  found to ensure we have all the requisite information to publish your
  device and avoid conflicts.

  Finally, sometimes we have to play "namespace police."  Please don't be
  offended.  We often get submissions for ``/dev`` names that would be bound
  to cause conflicts down the road.  We are trying to avoid getting in a
  situation where we would have to suffer an incompatible forward
  change.  Therefore, please consult with us **before** you make your
  device names and numbers in any way public, at least to the point
  where it would be at all difficult to get them changed.

  Your cooperation is appreciated.

.. include:: devices.txt
   :literal:

Additional ``/dev/`` directory entries
--------------------------------------

This section details additional entries that should or may exist in
the /dev directory.  It is preferred that symbolic links use the same
form (absolute or relative) as is indicated here.  Links are
classified as "hard" or "symbolic" depending on the preferred type of
link; if possible, the indicated type of link should be used.

Compulsory links
++++++++++++++++

These links should exist on all systems:

=============== =============== =============== ===============================
/dev/fd		/proc/self/fd	symbolic	File descriptors
/dev/stdin	fd/0		symbolic	stdin file descriptor
/dev/stdout	fd/1		symbolic	stdout file descriptor
/dev/stderr	fd/2		symbolic	stderr file descriptor
/dev/nfsd	socksys		symbolic	Required by iBCS-2
/dev/X0R	null		symbolic	Required by iBCS-2
=============== =============== =============== ===============================

Note: ``/dev/X0R`` is <letter X>-<digit 0>-<letter R>.

Recommended links
+++++++++++++++++

It is recommended that these links exist on all systems:

=============== =============== =============== ===============================
/dev/core	/proc/kcore	symbolic	Backward compatibility
/dev/ramdisk	ram0		symbolic	Backward compatibility
/dev/ftape	qft0		symbolic	Backward compatibility
/dev/bttv0	video0		symbolic	Backward compatibility
/dev/radio	radio0		symbolic	Backward compatibility
/dev/i2o*	/dev/i2o/*	symbolic	Backward compatibility
/dev/scd?	sr?		hard		Alternate SCSI CD-ROM name
=============== =============== =============== ===============================

Locally defined links
+++++++++++++++++++++

The following links may be established locally to conform to the
configuration of the system.  This is merely a tabulation of existing
practice, and does not constitute a recommendation.  However, if they
exist, they should have the following uses.

=============== =============== =============== ===============================
/dev/mouse	mouse port	symbolic	Current mouse device
/dev/tape	tape device	symbolic	Current tape device
/dev/cdrom	CD-ROM device	symbolic	Current CD-ROM device
/dev/cdwriter	CD-writer	symbolic	Current CD-writer device
/dev/scanner	scanner		symbolic	Current scanner device
/dev/modem	modem port	symbolic	Current dialout device
/dev/root	root device	symbolic	Current root filesystem
/dev/swap	swap device	symbolic	Current swap device
=============== =============== =============== ===============================

``/dev/modem`` should not be used for a modem which supports dialin as
well as dialout, as it tends to cause lock file problems.  If it
exists, ``/dev/modem`` should point to the appropriate primary TTY device
(the use of the alternate callout devices is deprecated).

For SCSI devices, ``/dev/tape`` and ``/dev/cdrom`` should point to the
*cooked* devices (``/dev/st*`` and ``/dev/sr*``, respectively), whereas
``/dev/cdwriter`` and /dev/scanner should point to the appropriate generic
SCSI devices (/dev/sg*).

``/dev/mouse`` may point to a primary serial TTY device, a hardware mouse
device, or a socket for a mouse driver program (e.g. ``/dev/gpmdata``).

Sockets and pipes
+++++++++++++++++

Non-transient sockets and named pipes may exist in /dev.  Common entries are:

=============== =============== ===============================================
/dev/printer	socket		lpd local socket
/dev/log	socket		syslog local socket
/dev/gpmdata	socket		gpm mouse multiplexer
=============== =============== ===============================================

Mount points
++++++++++++

The following names are reserved for mounting special filesystems
under /dev.  These special filesystems provide kernel interfaces that
cannot be provided with standard device nodes.

=============== =============== ===============================================
/dev/pts	devpts		PTY slave filesystem
/dev/shm	tmpfs		POSIX shared memory maintenance access
=============== =============== ===============================================

Terminal devices
----------------

Terminal, or TTY devices are a special class of character devices.  A
terminal device is any device that could act as a controlling terminal
for a session; this includes virtual consoles, serial ports, and
pseudoterminals (PTYs).

All terminal devices share a common set of capabilities known as line
disciplines; these include the common terminal line discipline as well
as SLIP and PPP modes.

All terminal devices are named similarly; this section explains the
naming and use of the various types of TTYs.  Note that the naming
conventions include several historical warts; some of these are
Linux-specific, some were inherited from other systems, and some
reflect Linux outgrowing a borrowed convention.

A hash mark (``#``) in a device name is used here to indicate a decimal
number without leading zeroes.

Virtual consoles and the console device
+++++++++++++++++++++++++++++++++++++++

Virtual consoles are full-screen terminal displays on the system video
monitor.  Virtual consoles are named ``/dev/tty#``, with numbering
starting at ``/dev/tty1``; ``/dev/tty0`` is the current virtual console.
``/dev/tty0`` is the device that should be used to access the system video
card on those architectures for which the frame buffer devices
(``/dev/fb*``) are not applicable. Do not use ``/dev/console``
for this purpose.

The console device, ``/dev/console``, is the device to which system
messages should be sent, and on which logins should be permitted in
single-user mode.  Starting with Linux 2.1.71, ``/dev/console`` is managed
by the kernel; for previous versions it should be a symbolic link to
either ``/dev/tty0``, a specific virtual console such as ``/dev/tty1``, or to
a serial port primary (``tty*``, not ``cu*``) device, depending on the
configuration of the system.

Serial ports
++++++++++++

Serial ports are RS-232 serial ports and any device which simulates
one, either in hardware (such as internal modems) or in software (such
as the ISDN driver.)  Under Linux, each serial ports has two device
names, the primary or callin device and the alternate or callout one.
Each kind of device is indicated by a different letter.	 For any
letter X, the names of the devices are ``/dev/ttyX#`` and ``/dev/cux#``,
respectively; for historical reasons, ``/dev/ttyS#`` and ``/dev/ttyC#``
correspond to ``/dev/cua#`` and ``/dev/cub#``. In the future, it should be
expected that multiple letters will be used; all letters will be upper
case for the "tty" device (e.g. ``/dev/ttyDP#``) and lower case for the
"cu" device (e.g. ``/dev/cudp#``).

The names ``/dev/ttyQ#`` and ``/dev/cuq#`` are reserved for local use.

The alternate devices provide for kernel-based exclusion and somewhat
different defaults than the primary devices.  Their main purpose is to
allow the use of serial ports with programs with no inherent or broken
support for serial ports.  Their use is deprecated, and they may be
removed from a future version of Linux.

Arbitration of serial ports is provided by the use of lock files with
the names ``/var/lock/LCK..ttyX#``. The contents of the lock file should
be the PID of the locking process as an ASCII number.

It is common practice to install links such as /dev/modem
which point to serial ports.  In order to ensure proper locking in the
presence of these links, it is recommended that software chase
symlinks and lock all possible names; additionally, it is recommended
that a lock file be installed with the corresponding alternate
device.	 In order to avoid deadlocks, it is recommended that the locks
are acquired in the following order, and released in the reverse:

	1. The symbolic link name, if any (``/var/lock/LCK..modem``)
	2. The "tty" name (``/var/lock/LCK..ttyS2``)
	3. The alternate device name (``/var/lock/LCK..cua2``)

In the case of nested symbolic links, the lock files should be
installed in the order the symlinks are resolved.

Under no circumstances should an application hold a lock while waiting
for another to be released.  In addition, applications which attempt
to create lock files for the corresponding alternate device names
should take into account the possibility of being used on a non-serial
port TTY, for which no alternate device would exist.

Pseudoterminals (PTYs)
++++++++++++++++++++++

Pseudoterminals, or PTYs, are used to create login sessions or provide
other capabilities requiring a TTY line discipline (including SLIP or
PPP capability) to arbitrary data-generation processes.	 Each PTY has
a master side, named ``/dev/pty[p-za-e][0-9a-f]``, and a slave side, named
``/dev/tty[p-za-e][0-9a-f]``.  The kernel arbitrates the use of PTYs by
allowing each master side to be opened only once.

Once the master side has been opened, the corresponding slave device
can be used in the same manner as any TTY device.  The master and
slave devices are connected by the kernel, generating the equivalent
of a bidirectional pipe with TTY capabilities.

Recent versions of the Linux kernels and GNU libc contain support for
the System V/Unix98 naming scheme for PTYs, which assigns a common
device, ``/dev/ptmx``, to all the masters (opening it will automatically
give you a previously unassigned PTY) and a subdirectory, ``/dev/pts``,
for the slaves; the slaves are named with decimal integers (``/dev/pts/#``
in our notation).  This removes the problem of exhausting the
namespace and enables the kernel to automatically create the device
nodes for the slaves on demand using the "devpts" filesystem.