code:
=======
Authors
=======

Original Author
---------------

Steve French (smfrench@gmail.com, sfrench@samba.org)

The author wishes to express his appreciation and thanks to:
Andrew Tridgell (Samba team) for his early suggestions about SMB/CIFS VFS
improvements. Thanks to IBM for allowing me time and test resources to pursue
this project, to Jim McDonough from IBM (and the Samba Team) for his help, to
the IBM Linux JFS team for explaining many esoteric Linux filesystem features.
Jeremy Allison of the Samba team has done invaluable work in adding the server
side of the original CIFS Unix extensions and reviewing and implementing
portions of the newer CIFS POSIX extensions into the Samba 3 file server. Thank
Dave Boutcher of IBM Rochester (author of the OS/400 smb/cifs filesystem client)
for proving years ago that very good smb/cifs clients could be done on Unix-like
operating systems.  Volker Lendecke, Andrew Tridgell, Urban Widmark, John
Newbigin and others for their work on the Linux smbfs module.  Thanks to
the other members of the Storage Network Industry Association CIFS Technical
Workgroup for their work specifying this highly complex protocol and finally
thanks to the Samba team for their technical advice and encouragement.

Patch Contributors
------------------

- Zwane Mwaikambo
- Andi Kleen
- Amrut Joshi
- Shobhit Dayal
- Sergey Vlasov
- Richard Hughes
- Yury Umanets
- Mark Hamzy (for some of the early cifs IPv6 work)
- Domen Puncer
- Jesper Juhl (in particular for lots of whitespace/formatting cleanup)
- Vince Negri and Dave Stahl (for finding an important caching bug)
- Adrian Bunk (kcalloc cleanups)
- Miklos Szeredi
- Kazeon team for various fixes especially for 2.4 version.
- Asser Ferno (Change Notify support)
- Shaggy (Dave Kleikamp) for innumerable small fs suggestions and some good cleanup
- Gunter Kukkukk (testing and suggestions for support of old servers)
- Igor Mammedov (DFS support)
- Jeff Layton (many, many fixes, as well as great work on the cifs Kerberos code)
- Scott Lovenberg
- Pavel Shilovsky (for great work adding SMB2 support, and various SMB3 features)
- Aurelien Aptel (for DFS SMB3 work and some key bug fixes)
- Ronnie Sahlberg (for SMB3 xattr work, bug fixes, and lots of great work on compounding)
- Shirish Pargaonkar (for many ACL patches over the years)
- Sachin Prabhu (many bug fixes, including for reconnect, copy offload and security)
- Paulo Alcantara (for some excellent work in DFS, and in booting from SMB3)
- Long Li (some great work on RDMA, SMB Direct)

Test case and Bug Report contributors
-------------------------------------
Thanks to those in the community who have submitted detailed bug reports
and debug of problems they have found:  Jochen Dolze, David Blaine,
Rene Scharfe, Martin Josefsson, Alexander Wild, Anthony Liguori,
Lars Muller, Urban Widmark, Massimiliano Ferrero, Howard Owen,
Olaf Kirch, Kieron Briggs, Nick Millington and others. Also special
mention to the Stanford Checker (SWAT) which pointed out many minor
bugs in error paths.  Valuable suggestions also have come from Al Viro
and Dave Miller.

And thanks to the IBM LTC and Power test teams and SuSE and Citrix and RedHat testers for finding multiple bugs during excellent stress test runs.

repo_name: angery/linux


code:
=======
Changes
=======

See https://wiki.samba.org/index.php/LinuxCIFSKernel for summary
information about fixes/improvements to CIFS/SMB2/SMB3 support (changes
to cifs.ko module) by kernel version (and cifs internal module version).
This may be easier to read than parsing the output of "git log fs/cifs"
by release.


code:
.. SPDX-License-Identifier: GPL-2.0

====
CIFS
====

.. toctree::
   :maxdepth: 2

   introduction
   usage
   todo
   changes
   authors

.. only::  subproject and html

   Indices
   =======

   * :ref:`genindex`


code:
============
Introduction
============

  This is the client VFS module for the SMB3 NAS protocol as well
  as for older dialects such as the Common Internet File System (CIFS)
  protocol which was the successor to the Server Message Block
  (SMB) protocol, the native file sharing mechanism for most early
  PC operating systems. New and improved versions of CIFS are now
  called SMB2 and SMB3. Use of SMB3 (and later, including SMB3.1.1
  the most current dialect) is strongly preferred over using older
  dialects like CIFS due to security reasons. All modern dialects,
  including the most recent, SMB3.1.1, are supported by the CIFS VFS
  module. The SMB3 protocol is implemented and supported by all major
  file servers such as Windows (including Windows 2019 Server), as
  well as by Samba (which provides excellent CIFS/SMB2/SMB3 server
  support and tools for Linux and many other operating systems).
  Apple systems also support SMB3 well, as do most Network Attached
  Storage vendors, so this network filesystem client can mount to a
  wide variety of systems. It also supports mounting to the cloud
  (for example Microsoft Azure), including the necessary security
  features.

  The intent of this module is to provide the most advanced network
  file system function for SMB3 compliant servers, including advanced
  security features, excellent parallelized high performance i/o, better
  POSIX compliance, secure per-user session establishment, encryption,
  high performance safe distributed caching (leases/oplocks), optional packet
  signing, large files, Unicode support and other internationalization
  improvements. Since both Samba server and this filesystem client support the
  CIFS Unix extensions, and the Linux client also suppors SMB3 POSIX extensions,
  the combination can provide a reasonable alternative to other network and
  cluster file systems for fileserving in some Linux to Linux environments,
  not just in Linux to Windows (or Linux to Mac) environments.

  This filesystem has a mount utility (mount.cifs) and various user space
  tools (including smbinfo and setcifsacl) that can be obtained from

      https://git.samba.org/?p=cifs-utils.git

  or

      git://git.samba.org/cifs-utils.git

  mount.cifs should be installed in the directory with the other mount helpers.

  For more information on the module see the project wiki page at

      https://wiki.samba.org/index.php/LinuxCIFS

  and

      https://wiki.samba.org/index.php/LinuxCIFS_utils


code:
====
TODO
====

Version 2.14 December 21, 2018

A Partial List of Missing Features
==================================

Contributions are welcome.  There are plenty of opportunities
for visible, important contributions to this module.  Here
is a partial list of the known problems and missing features:

a) SMB3 (and SMB3.1.1) missing optional features:

   - multichannel (partially integrated), integration of multichannel with RDMA
   - directory leases (improved metadata caching). Currently only implemented for root dir
   - T10 copy offload ie "ODX" (copy chunk, and "Duplicate Extents" ioctl
     currently the only two server side copy mechanisms supported)

b) improved sparse file support (fiemap and SEEK_HOLE are implemented
   but additional features would be supportable by the protocol such
   as FALLOC_FL_COLLAPSE_RANGE and FALLOC_FL_INSERT_RANGE)

c) Directory entry caching relies on a 1 second timer, rather than
   using Directory Leases, currently only the root file handle is cached longer
   by leveraging Directory Leases

d) quota support (needs minor kernel change since quota calls otherwise
    won't make it to network filesystems or deviceless filesystems).

e) Additional use cases can be optimized to use "compounding" (e.g.
   open/query/close and open/setinfo/close) to reduce the number of
   roundtrips to the server and improve performance. Various cases
   (stat, statfs, create, unlink, mkdir, xattrs) already have been improved by
   using compounding but more can be done. In addition we could
   significantly reduce redundant opens by using deferred close (with
   handle caching leases) and better using reference counters on file
   handles.

f) Finish inotify support so kde and gnome file list windows
   will autorefresh (partially complete by Asser). Needs minor kernel
   vfs change to support removing D_NOTIFY on a file.

g) Add GUI tool to configure /proc/fs/cifs settings and for display of
   the CIFS statistics (started)

h) implement support for security and trusted categories of xattrs
   (requires minor protocol extension) to enable better support for SELINUX

i) Add support for tree connect contexts (see MS-SMB2) a new SMB3.1.1 protocol
   feature (may be especially useful for virtualization).

j) Create UID mapping facility so server UIDs can be mapped on a per
   mount or a per server basis to client UIDs or nobody if no mapping
   exists. Also better integration with winbind for resolving SID owners

k) Add tools to take advantage of more smb3 specific ioctls and features
   (passthrough ioctl/fsctl is now implemented in cifs.ko to allow
   sending various SMB3 fsctls and query info and set info calls
   directly from user space) Add tools to make setting various non-POSIX
   metadata attributes easier from tools (e.g. extending what was done
   in smb-info tool).

l) encrypted file support (currently the attribute showing the file is
   encrypted on the server is reported, but changing the attribute is not
   supported).

m) improved stats gathering tools (perhaps integration with nfsometer?)
   to extend and make easier to use what is currently in /proc/fs/cifs/Stats

n) Add support for claims based ACLs ("DAC")

o) mount helper GUI (to simplify the various configuration options on mount)

p) Expand support for witness protocol to allow for notification of share
   move, and server network adapter changes. Currently only notifications by
   the witness protocol for server move is supported by the Linux client.

q) Allow mount.cifs to be more verbose in reporting errors with dialect
   or unsupported feature errors. This would now be easier due to the
   implementation of the new mount API.

r) updating cifs documentation, and user guide.

s) Addressing bugs found by running a broader set of xfstests in standard
   file system xfstest suite.

t) split cifs and smb3 support into separate modules so legacy (and less
   secure) CIFS dialect can be disabled in environments that don't need it
   and simplify the code.

v) Additional testing of POSIX Extensions for SMB3.1.1

w) Add support for additional strong encryption types, and additional spnego
   authentication mechanisms (see MS-SMB2).  GCM-256 is now partially implemented.

x) Finish support for SMB3.1.1 compression

Known Bugs
==========

See https://bugzilla.samba.org - search on product "CifsVFS" for
current bug list.  Also check http://bugzilla.kernel.org (Product = File System, Component = CIFS)

1) existing symbolic links (Windows reparse points) are recognized but
   can not be created remotely. They are implemented for Samba and those that
   support the CIFS Unix extensions, although earlier versions of Samba
   overly restrict the pathnames.
2) follow_link and readdir code does not follow dfs junctions
   but recognizes them

Misc testing to do
==================
1) check out max path names and max path name components against various server
   types. Try nested symlinks (8 deep). Return max path name in stat -f information

2) Improve xfstest's cifs/smb3 enablement and adapt xfstests where needed to test
   cifs/smb3 better

3) Additional performance testing and optimization using iozone and similar -
   there are some easy changes that can be done to parallelize sequential writes,
   and when signing is disabled to request larger read sizes (larger than
   negotiated size) and send larger write sizes to modern servers.

4) More exhaustively test against less common servers

5) Continue to extend the smb3 "buildbot" which does automated xfstesting
   against Windows, Samba and Azure currently - to add additional tests and
   to allow the buildbot to execute the tests faster. The URL for the
   buildbot is: http://smb3-test-rhel-75.southcentralus.cloudapp.azure.com

6) Address various coverity warnings (most are not bugs per-se, but
   the more warnings are addressed, the easier it is to spot real
   problems that static analyzers will point out in the future).


code:
=====
Usage
=====

This module supports the SMB3 family of advanced network protocols (as well
as older dialects, originally called "CIFS" or SMB1).

The CIFS VFS module for Linux supports many advanced network filesystem
features such as hierarchical DFS like namespace, hardlinks, locking and more.
It was designed to comply with the SNIA CIFS Technical Reference (which
supersedes the 1992 X/Open SMB Standard) as well as to perform best practice
practical interoperability with Windows 2000, Windows XP, Samba and equivalent
servers.  This code was developed in participation with the Protocol Freedom
Information Foundation.  CIFS and now SMB3 has now become a defacto
standard for interoperating between Macs and Windows and major NAS appliances.

Please see
MS-SMB2 (for detailed SMB2/SMB3/SMB3.1.1 protocol specification)
or https://samba.org/samba/PFIF/
for more details.

For questions or bug reports please contact:

    smfrench@gmail.com

See the project page at: https://wiki.samba.org/index.php/LinuxCIFS_utils

Build instructions
==================

For Linux:

1) Download the kernel (e.g. from https://www.kernel.org)
   and change directory into the top of the kernel directory tree
   (e.g. /usr/src/linux-2.5.73)
2) make menuconfig (or make xconfig)
3) select cifs from within the network filesystem choices
4) save and exit
5) make

Installation instructions
=========================

If you have built the CIFS vfs as module (successfully) simply
type ``make modules_install`` (or if you prefer, manually copy the file to
the modules directory e.g. /lib/modules/2.4.10-4GB/kernel/fs/cifs/cifs.ko).

If you have built the CIFS vfs into the kernel itself, follow the instructions
for your distribution on how to install a new kernel (usually you
would simply type ``make install``).

If you do not have the utility mount.cifs (in the Samba 4.x source tree and on
the CIFS VFS web site) copy it to the same directory in which mount helpers
reside (usually /sbin).  Although the helper software is not
required, mount.cifs is recommended.  Most distros include a ``cifs-utils``
package that includes this utility so it is recommended to install this.

Note that running the Winbind pam/nss module (logon service) on all of your
Linux clients is useful in mapping Uids and Gids consistently across the
domain to the proper network user.  The mount.cifs mount helper can be
found at cifs-utils.git on git.samba.org

If cifs is built as a module, then the size and number of network buffers
and maximum number of simultaneous requests to one server can be configured.
Changing these from their defaults is not recommended. By executing modinfo::

	modinfo kernel/fs/cifs/cifs.ko

on kernel/fs/cifs/cifs.ko the list of configuration changes that can be made
at module initialization time (by running insmod cifs.ko) can be seen.

Recommendations
===============

To improve security the SMB2.1 dialect or later (usually will get SMB3) is now
the new default. To use old dialects (e.g. to mount Windows XP) use "vers=1.0"
on mount (or vers=2.0 for Windows Vista).  Note that the CIFS (vers=1.0) is
much older and less secure than the default dialect SMB3 which includes
many advanced security features such as downgrade attack detection
and encrypted shares and stronger signing and authentication algorithms.
There are additional mount options that may be helpful for SMB3 to get
improved POSIX behavior (NB: can use vers=3.0 to force only SMB3, never 2.1):

   ``mfsymlinks`` and either ``cifsacl`` or ``modefromsid`` (usually with ``idsfromsid``)

Allowing User Mounts
====================

To permit users to mount and unmount over directories they own is possible
with the cifs vfs.  A way to enable such mounting is to mark the mount.cifs
utility as suid (e.g. ``chmod +s /sbin/mount.cifs``). To enable users to
umount shares they mount requires

1) mount.cifs version 1.4 or later
2) an entry for the share in /etc/fstab indicating that a user may
   unmount it e.g.::

     //server/usersharename  /mnt/username cifs user 0 0

Note that when the mount.cifs utility is run suid (allowing user mounts),
in order to reduce risks, the ``nosuid`` mount flag is passed in on mount to
disallow execution of an suid program mounted on the remote target.
When mount is executed as root, nosuid is not passed in by default,
and execution of suid programs on the remote target would be enabled
by default. This can be changed, as with nfs and other filesystems,
by simply specifying ``nosuid`` among the mount options. For user mounts
though to be able to pass the suid flag to mount requires rebuilding
mount.cifs with the following flag: CIFS_ALLOW_USR_SUID

There is a corresponding manual page for cifs mounting in the Samba 3.0 and
later source tree in docs/manpages/mount.cifs.8

Allowing User Unmounts
======================

To permit users to unmount directories that they have user mounted (see above),
the utility umount.cifs may be used.  It may be invoked directly, or if
umount.cifs is placed in /sbin, umount can invoke the cifs umount helper
(at least for most versions of the umount utility) for umount of cifs
mounts, unless umount is invoked with -i (which will avoid invoking a umount
helper). As with mount.cifs, to enable user unmounts umount.cifs must be marked
as suid (e.g. ``chmod +s /sbin/umount.cifs``) or equivalent (some distributions
allow adding entries to a file to the /etc/permissions file to achieve the
equivalent suid effect).  For this utility to succeed the target path
must be a cifs mount, and the uid of the current user must match the uid
of the user who mounted the resource.

Also note that the customary way of allowing user mounts and unmounts is
(instead of using mount.cifs and unmount.cifs as suid) to add a line
to the file /etc/fstab for each //server/share you wish to mount, but
this can become unwieldy when potential mount targets include many
or  unpredictable UNC names.

Samba Considerations
====================

Most current servers support SMB2.1 and SMB3 which are more secure,
but there are useful protocol extensions for the older less secure CIFS
dialect, so to get the maximum benefit if mounting using the older dialect
(CIFS/SMB1), we recommend using a server that supports the SNIA CIFS
Unix Extensions standard (e.g. almost any  version of Samba ie version
2.2.5 or later) but the CIFS vfs works fine with a wide variety of CIFS servers.
Note that uid, gid and file permissions will display default values if you do
not have a server that supports the Unix extensions for CIFS (such as Samba
2.2.5 or later).  To enable the Unix CIFS Extensions in the Samba server, add
the line::

	unix extensions = yes

to your smb.conf file on the server.  Note that the following smb.conf settings
are also useful (on the Samba server) when the majority of clients are Unix or
Linux::

	case sensitive = yes
	delete readonly = yes
	ea support = yes

Note that server ea support is required for supporting xattrs from the Linux
cifs client, and that EA support is present in later versions of Samba (e.g.
3.0.6 and later (also EA support works in all versions of Windows, at least to
shares on NTFS filesystems).  Extended Attribute (xattr) support is an optional
feature of most Linux filesystems which may require enabling via
make menuconfig. Client support for extended attributes (user xattr) can be
disabled on a per-mount basis by specifying ``nouser_xattr`` on mount.

The CIFS client can get and set POSIX ACLs (getfacl, setfacl) to Samba servers
version 3.10 and later.  Setting POSIX ACLs requires enabling both XATTR and
then POSIX support in the CIFS configuration options when building the cifs
module.  POSIX ACL support can be disabled on a per mount basic by specifying
``noacl`` on mount.

Some administrators may want to change Samba's smb.conf ``map archive`` and
``create mask`` parameters from the default.  Unless the create mask is changed
newly created files can end up with an unnecessarily restrictive default mode,
which may not be what you want, although if the CIFS Unix extensions are
enabled on the server and client, subsequent setattr calls (e.g. chmod) can
fix the mode.  Note that creating special devices (mknod) remotely
may require specifying a mkdev function to Samba if you are not using
Samba 3.0.6 or later.  For more information on these see the manual pages
(``man smb.conf``) on the Samba server system.  Note that the cifs vfs,
unlike the smbfs vfs, does not read the smb.conf on the client system
(the few optional settings are passed in on mount via -o parameters instead).
Note that Samba 2.2.7 or later includes a fix that allows the CIFS VFS to delete
open files (required for strict POSIX compliance).  Windows Servers already
supported this feature. Samba server does not allow symlinks that refer to files
outside of the share, so in Samba versions prior to 3.0.6, most symlinks to
files with absolute paths (ie beginning with slash) such as::

	 ln -s /mnt/foo bar

would be forbidden. Samba 3.0.6 server or later includes the ability to create
such symlinks safely by converting unsafe symlinks (ie symlinks to server
files that are outside of the share) to a samba specific format on the server
that is ignored by local server applications and non-cifs clients and that will
not be traversed by the Samba server).  This is opaque to the Linux client
application using the cifs vfs. Absolute symlinks will work to Samba 3.0.5 or
later, but only for remote clients using the CIFS Unix extensions, and will
be invisible to Windows clients and typically will not affect local
applications running on the same server as Samba.

Use instructions
================

Once the CIFS VFS support is built into the kernel or installed as a module
(cifs.ko), you can use mount syntax like the following to access Samba or
Mac or Windows servers::

  mount -t cifs //9.53.216.11/e$ /mnt -o username=myname,password=mypassword

Before -o the option -v may be specified to make the mount.cifs
mount helper display the mount steps more verbosely.
After -o the following commonly used cifs vfs specific options
are supported::

  username=<username>
  password=<password>
  domain=<domain name>

Other cifs mount options are described below.  Use of TCP names (in addition to
ip addresses) is available if the mount helper (mount.cifs) is installed. If
you do not trust the server to which are mounted, or if you do not have
cifs signing enabled (and the physical network is insecure), consider use
of the standard mount options ``noexec`` and ``nosuid`` to reduce the risk of
running an altered binary on your local system (downloaded from a hostile server
or altered by a hostile router).

Although mounting using format corresponding to the CIFS URL specification is
not possible in mount.cifs yet, it is possible to use an alternate format
for the server and sharename (which is somewhat similar to NFS style mount
syntax) instead of the more widely used UNC format (i.e. \\server\share)::

  mount -t cifs tcp_name_of_server:share_name /mnt -o user=myname,pass=mypasswd

When using the mount helper mount.cifs, passwords may be specified via alternate
mechanisms, instead of specifying it after -o using the normal ``pass=`` syntax
on the command line:
1) By including it in a credential file. Specify credentials=filename as one
of the mount options. Credential files contain two lines::

	username=someuser
	password=your_password

2) By specifying the password in the PASSWD environment variable (similarly
   the user name can be taken from the USER environment variable).
3) By specifying the password in a file by name via PASSWD_FILE
4) By specifying the password in a file by file descriptor via PASSWD_FD

If no password is provided, mount.cifs will prompt for password entry

Restrictions
============

Servers must support either "pure-TCP" (port 445 TCP/IP CIFS connections) or RFC
1001/1002 support for "Netbios-Over-TCP/IP." This is not likely to be a
problem as most servers support this.

Valid filenames differ between Windows and Linux.  Windows typically restricts
filenames which contain certain reserved characters (e.g.the character :
which is used to delimit the beginning of a stream name by Windows), while
Linux allows a slightly wider set of valid characters in filenames. Windows
servers can remap such characters when an explicit mapping is specified in
the Server's registry.  Samba starting with version 3.10 will allow such
filenames (ie those which contain valid Linux characters, which normally
would be forbidden for Windows/CIFS semantics) as long as the server is
configured for Unix Extensions (and the client has not disabled
/proc/fs/cifs/LinuxExtensionsEnabled). In addition the mount option
``mapposix`` can be used on CIFS (vers=1.0) to force the mapping of
illegal Windows/NTFS/SMB characters to a remap range (this mount parameter
is the default for SMB3). This remap (``mapposix``) range is also
compatible with Mac (and "Services for Mac" on some older Windows).

CIFS VFS Mount Options
======================
A partial list of the supported mount options follows:

  username
		The user name to use when trying to establish
		the CIFS session.
  password
		The user password.  If the mount helper is
		installed, the user will be prompted for password
		if not supplied.
  ip
		The ip address of the target server
  unc
		The target server Universal Network Name (export) to
		mount.
  domain
		Set the SMB/CIFS workgroup name prepended to the
		username during CIFS session establishment
  forceuid
		Set the default uid for inodes to the uid
		passed in on mount. For mounts to servers
		which do support the CIFS Unix extensions, such as a
		properly configured Samba server, the server provides
		the uid, gid and mode so this parameter should not be
		specified unless the server and clients uid and gid
		numbering differ.  If the server and client are in the
		same domain (e.g. running winbind or nss_ldap) and
		the server supports the Unix Extensions then the uid
		and gid can be retrieved from the server (and uid
		and gid would not have to be specified on the mount.
		For servers which do not support the CIFS Unix
		extensions, the default uid (and gid) returned on lookup
		of existing files will be the uid (gid) of the person
		who executed the mount (root, except when mount.cifs
		is configured setuid for user mounts) unless the ``uid=``
		(gid) mount option is specified. Also note that permission
		checks (authorization checks) on accesses to a file occur
		at the server, but there are cases in which an administrator
		may want to restrict at the client as well.  For those
		servers which do not report a uid/gid owner
		(such as Windows), permissions can also be checked at the
		client, and a crude form of client side permission checking
		can be enabled by specifying file_mode and dir_mode on
		the client.  (default)
  forcegid
		(similar to above but for the groupid instead of uid) (default)
  noforceuid
		Fill in file owner information (uid) by requesting it from
		the server if possible. With this option, the value given in
		the uid= option (on mount) will only be used if the server
		can not support returning uids on inodes.
  noforcegid
		(similar to above but for the group owner, gid, instead of uid)
  uid
		Set the default uid for inodes, and indicate to the
		cifs kernel driver which local user mounted. If the server
		supports the unix extensions the default uid is
		not used to fill in the owner fields of inodes (files)
		unless the ``forceuid`` parameter is specified.
  gid
		Set the default gid for inodes (similar to above).
  file_mode
		If CIFS Unix extensions are not supported by the server
		this overrides the default mode for file inodes.
  fsc
		Enable local disk caching using FS-Cache (off by default). This
		option could be useful to improve performance on a slow link,
		heavily loaded server and/or network where reading from the
		disk is faster than reading from the server (over the network).
		This could also impact scalability positively as the
		number of calls to the server are reduced. However, local
		caching is not suitable for all workloads for e.g. read-once
		type workloads. So, you need to consider carefully your
		workload/scenario before using this option. Currently, local
		disk caching is functional for CIFS files opened as read-only.
  dir_mode
		If CIFS Unix extensions are not supported by the server
		this overrides the default mode for directory inodes.
  port
		attempt to contact the server on this tcp port, before
		trying the usual ports (port 445, then 139).
  iocharset
		Codepage used to convert local path names to and from
		Unicode. Unicode is used by default for network path
		names if the server supports it.  If iocharset is
		not specified then the nls_default specified
		during the local client kernel build will be used.
		If server does not support Unicode, this parameter is
		unused.
  rsize
		default read size (usually 16K). The client currently
		can not use rsize larger than CIFSMaxBufSize. CIFSMaxBufSize
		defaults to 16K and may be changed (from 8K to the maximum
		kmalloc size allowed by your kernel) at module install time
		for cifs.ko. Setting CIFSMaxBufSize to a very large value
		will cause cifs to use more memory and may reduce performance
		in some cases.  To use rsize greater than 127K (the original
		cifs protocol maximum) also requires that the server support
		a new Unix Capability flag (for very large read) which some
		newer servers (e.g. Samba 3.0.26 or later) do. rsize can be
		set from a minimum of 2048 to a maximum of 130048 (127K or
		CIFSMaxBufSize, whichever is smaller)
  wsize
		default write size (default 57344)
		maximum wsize currently allowed by CIFS is 57344 (fourteen
		4096 byte pages)
  actimeo=n
		attribute cache timeout in seconds (default 1 second).
		After this timeout, the cifs client requests fresh attribute
		information from the server. This option allows to tune the
		attribute cache timeout to suit the workload needs. Shorter
		timeouts mean better the cache coherency, but increased number
		of calls to the server. Longer timeouts mean reduced number
		of calls to the server at the expense of less stricter cache
		coherency checks (i.e. incorrect attribute cache for a short
		period of time).
  rw
		mount the network share read-write (note that the
		server may still consider the share read-only)
  ro
		mount network share read-only
  version
		used to distinguish different versions of the
		mount helper utility (not typically needed)
  sep
		if first mount option (after the -o), overrides
		the comma as the separator between the mount
		parms. e.g.::

			-o user=myname,password=mypassword,domain=mydom

		could be passed instead with period as the separator by::

			-o sep=.user=myname.password=mypassword.domain=mydom

		this might be useful when comma is contained within username
		or password or domain. This option is less important
		when the cifs mount helper cifs.mount (version 1.1 or later)
		is used.
  nosuid
		Do not allow remote executables with the suid bit
		program to be executed.  This is only meaningful for mounts
		to servers such as Samba which support the CIFS Unix Extensions.
		If you do not trust the servers in your network (your mount
		targets) it is recommended that you specify this option for
		greater security.
  exec
		Permit execution of binaries on the mount.
  noexec
		Do not permit execution of binaries on the mount.
  dev
		Recognize block devices on the remote mount.
  nodev
		Do not recognize devices on the remote mount.
  suid
		Allow remote files on this mountpoint with suid enabled to
		be executed (default for mounts when executed as root,
		nosuid is default for user mounts).
  credentials
		Although ignored by the cifs kernel component, it is used by
		the mount helper, mount.cifs. When mount.cifs is installed it
		opens and reads the credential file specified in order
		to obtain the userid and password arguments which are passed to
		the cifs vfs.
  guest
		Although ignored by the kernel component, the mount.cifs
		mount helper will not prompt the user for a password
		if guest is specified on the mount options.  If no
		password is specified a null password will be used.
  perm
		Client does permission checks (vfs_permission check of uid
		and gid of the file against the mode and desired operation),
		Note that this is in addition to the normal ACL check on the
		target machine done by the server software.
		Client permission checking is enabled by default.
  noperm
		Client does not do permission checks.  This can expose
		files on this mount to access by other users on the local
		client system. It is typically only needed when the server
		supports the CIFS Unix Extensions but the UIDs/GIDs on the
		client and server system do not match closely enough to allow
		access by the user doing the mount, but it may be useful with
		non CIFS Unix Extension mounts for cases in which the default
		mode is specified on the mount but is not to be enforced on the
		client (e.g. perhaps when MultiUserMount is enabled)
		Note that this does not affect the normal ACL check on the
		target machine done by the server software (of the server
		ACL against the user name provided at mount time).
  serverino
		Use server's inode numbers instead of generating automatically
		incrementing inode numbers on the client.  Although this will
		make it easier to spot hardlinked files (as they will have
		the same inode numbers) and inode numbers may be persistent,
		note that the server does not guarantee that the inode numbers
		are unique if multiple server side mounts are exported under a
		single share (since inode numbers on the servers might not
		be unique if multiple filesystems are mounted under the same
		shared higher level directory).  Note that some older
		(e.g. pre-Windows 2000) do not support returning UniqueIDs
		or the CIFS Unix Extensions equivalent and for those
		this mount option will have no effect.  Exporting cifs mounts
		under nfsd requires this mount option on the cifs mount.
		This is now the default if server supports the
		required network operation.
  noserverino
		Client generates inode numbers (rather than using the actual one
		from the server). These inode numbers will vary after
		unmount or reboot which can confuse some applications,
		but not all server filesystems support unique inode
		numbers.
  setuids
		If the CIFS Unix extensions are negotiated with the server
		the client will attempt to set the effective uid and gid of
		the local process on newly created files, directories, and
		devices (create, mkdir, mknod).  If the CIFS Unix Extensions
		are not negotiated, for newly created files and directories
		instead of using the default uid and gid specified on
		the mount, cache the new file's uid and gid locally which means
		that the uid for the file can change when the inode is
		reloaded (or the user remounts the share).
  nosetuids
		The client will not attempt to set the uid and gid on
		on newly created files, directories, and devices (create,
		mkdir, mknod) which will result in the server setting the
		uid and gid to the default (usually the server uid of the
		user who mounted the share).  Letting the server (rather than
		the client) set the uid and gid is the default. If the CIFS
		Unix Extensions are not negotiated then the uid and gid for
		new files will appear to be the uid (gid) of the mounter or the
		uid (gid) parameter specified on the mount.
  netbiosname
		When mounting to servers via port 139, specifies the RFC1001
		source name to use to represent the client netbios machine
		name when doing the RFC1001 netbios session initialize.
  direct
		Do not do inode data caching on files opened on this mount.
		This precludes mmapping files on this mount. In some cases
		with fast networks and little or no caching benefits on the
		client (e.g. when the application is doing large sequential
		reads bigger than page size without rereading the same data)
		this can provide better performance than the default
		behavior which caches reads (readahead) and writes
		(writebehind) through the local Linux client pagecache
		if oplock (caching token) is granted and held. Note that
		direct allows write operations larger than page size
		to be sent to the server.
  strictcache
		Use for switching on strict cache mode. In this mode the
		client read from the cache all the time it has Oplock Level II,
		otherwise - read from the server. All written data are stored
		in the cache, but if the client doesn't have Exclusive Oplock,
		it writes the data to the server.
  rwpidforward
		Forward pid of a process who opened a file to any read or write
		operation on that file. This prevent applications like WINE
		from failing on read and write if we use mandatory brlock style.
  acl
		Allow setfacl and getfacl to manage posix ACLs if server
		supports them.  (default)
  noacl
		Do not allow setfacl and getfacl calls on this mount
  user_xattr
		Allow getting and setting user xattrs (those attributes whose
		name begins with ``user.`` or ``os2.``) as OS/2 EAs (extended
		attributes) to the server.  This allows support of the
		setfattr and getfattr utilities. (default)
  nouser_xattr
		Do not allow getfattr/setfattr to get/set/list xattrs
  mapchars
		Translate six of the seven reserved characters (not backslash)::

			*?<>|:

		to the remap range (above 0xF000), which also
		allows the CIFS client to recognize files created with
		such characters by Windows's POSIX emulation. This can
		also be useful when mounting to most versions of Samba
		(which also forbids creating and opening files
		whose names contain any of these seven characters).
		This has no effect if the server does not support
		Unicode on the wire.
  nomapchars
		Do not translate any of these seven characters (default).
  nocase
		Request case insensitive path name matching (case
		sensitive is the default if the server supports it).
		(mount option ``ignorecase`` is identical to ``nocase``)
  posixpaths
		If CIFS Unix extensions are supported, attempt to
		negotiate posix path name support which allows certain
		characters forbidden in typical CIFS filenames, without
		requiring remapping. (default)
  noposixpaths
		If CIFS Unix extensions are supported, do not request
		posix path name support (this may cause servers to
		reject creatingfile with certain reserved characters).
  nounix
		Disable the CIFS Unix Extensions for this mount (tree
		connection). This is rarely needed, but it may be useful
		in order to turn off multiple settings all at once (ie
		posix acls, posix locks, posix paths, symlink support
		and retrieving uids/gids/mode from the server) or to
		work around a bug in server which implement the Unix
		Extensions.
  nobrl
		Do not send byte range lock requests to the server.
		This is necessary for certain applications that break
		with cifs style mandatory byte range locks (and most
		cifs servers do not yet support requesting advisory
		byte range locks).
  forcemandatorylock
		Even if the server supports posix (advisory) byte range
		locking, send only mandatory lock requests.  For some
		(presumably rare) applications, originally coded for
		DOS/Windows, which require Windows style mandatory byte range
		locking, they may be able to take advantage of this option,
		forcing the cifs client to only send mandatory locks
		even if the cifs server would support posix advisory locks.
		``forcemand`` is accepted as a shorter form of this mount
		option.
  nostrictsync
		If this mount option is set, when an application does an
		fsync call then the cifs client does not send an SMB Flush
		to the server (to force the server to write all dirty data
		for this file immediately to disk), although cifs still sends
		all dirty (cached) file data to the server and waits for the
		server to respond to the write.  Since SMB Flush can be
		very slow, and some servers may be reliable enough (to risk
		delaying slightly flushing the data to disk on the server),
		turning on this option may be useful to improve performance for
		applications that fsync too much, at a small risk of server
		crash.  If this mount option is not set, by default cifs will
		send an SMB flush request (and wait for a response) on every
		fsync call.
  nodfs
		Disable DFS (global name space support) even if the
		server claims to support it.  This can help work around
		a problem with parsing of DFS paths with Samba server
		versions 3.0.24 and 3.0.25.
  remount
		remount the share (often used to change from ro to rw mounts
		or vice versa)
  cifsacl
		Report mode bits (e.g. on stat) based on the Windows ACL for
		the file. (EXPERIMENTAL)
  servern
		Specify the server 's netbios name (RFC1001 name) to use
		when attempting to setup a session to the server.
		This is needed for mounting to some older servers (such
		as OS/2 or Windows 98 and Windows ME) since they do not
		support a default server name.  A server name can be up
		to 15 characters long and is usually uppercased.
  sfu
		When the CIFS Unix Extensions are not negotiated, attempt to
		create device files and fifos in a format compatible with
		Services for Unix (SFU).  In addition retrieve bits 10-12
		of the mode via the SETFILEBITS extended attribute (as
		SFU does).  In the future the bottom 9 bits of the
		mode also will be emulated using queries of the security
		descriptor (ACL).
  mfsymlinks
		Enable support for Minshall+French symlinks
		(see http://wiki.samba.org/index.php/UNIX_Extensions#Minshall.2BFrench_symlinks)
		This option is ignored when specified together with the
		'sfu' option. Minshall+French symlinks are used even if
		the server supports the CIFS Unix Extensions.
  sign
		Must use packet signing (helps avoid unwanted data modification
		by intermediate systems in the route).  Note that signing
		does not work with lanman or plaintext authentication.
  seal
		Must seal (encrypt) all data on this mounted share before
		sending on the network.  Requires support for Unix Extensions.
		Note that this differs from the sign mount option in that it
		causes encryption of data sent over this mounted share but other
		shares mounted to the same server are unaffected.
  locallease
		This option is rarely needed. Fcntl F_SETLEASE is
		used by some applications such as Samba and NFSv4 server to
		check to see whether a file is cacheable.  CIFS has no way
		to explicitly request a lease, but can check whether a file
		is cacheable (oplocked).  Unfortunately, even if a file
		is not oplocked, it could still be cacheable (ie cifs client
		could grant fcntl leases if no other local processes are using
		the file) for cases for example such as when the server does not
		support oplocks and the user is sure that the only updates to
		the file will be from this client. Specifying this mount option
		will allow the cifs client to check for leases (only) locally
		for files which are not oplocked instead of denying leases
		in that case. (EXPERIMENTAL)
  sec
		Security mode.  Allowed values are:

			none
				attempt to connection as a null user (no name)
			krb5
				Use Kerberos version 5 authentication
			krb5i
				Use Kerberos authentication and packet signing
			ntlm
				Use NTLM password hashing (default)
			ntlmi
				Use NTLM password hashing with signing (if
				/proc/fs/cifs/PacketSigningEnabled on or if
				server requires signing also can be the default)
			ntlmv2
				Use NTLMv2 password hashing
			ntlmv2i
				Use NTLMv2 password hashing with packet signing
			lanman
				(if configured in kernel config) use older
				lanman hash
  hard
		Retry file operations if server is not responding
  soft
		Limit retries to unresponsive servers (usually only
		one retry) before returning an error.  (default)

The mount.cifs mount helper also accepts a few mount options before -o
including:

=============== ===============================================================
	-S      take password from stdin (equivalent to setting the environment
		variable ``PASSWD_FD=0``
	-V      print mount.cifs version
	-?      display simple usage information
=============== ===============================================================

With most 2.6 kernel versions of modutils, the version of the cifs kernel
module can be displayed via modinfo.

Misc /proc/fs/cifs Flags and Debug Info
=======================================

Informational pseudo-files:

======================= =======================================================
DebugData		Displays information about active CIFS sessions and
			shares, features enabled as well as the cifs.ko
			version.
Stats			Lists summary resource usage information as well as per
			share statistics.
open_files		List all the open file handles on all active SMB sessions.
======================= =======================================================

Configuration pseudo-files:

======================= =======================================================
SecurityFlags		Flags which control security negotiation and
			also packet signing. Authentication (may/must)
			flags (e.g. for NTLM and/or NTLMv2) may be combined with
			the signing flags.  Specifying two different password
			hashing mechanisms (as "must use") on the other hand
			does not make much sense. Default flags are::

				0x07007

			(NTLM, NTLMv2 and packet signing allowed).  The maximum
			allowable flags if you want to allow mounts to servers
			using weaker password hashes is 0x37037 (lanman,
			plaintext, ntlm, ntlmv2, signing allowed).  Some
			SecurityFlags require the corresponding menuconfig
			options to be enabled.  Enabling plaintext
			authentication currently requires also enabling
			lanman authentication in the security flags
			because the cifs module only supports sending
			laintext passwords using the older lanman dialect
			form of the session setup SMB.  (e.g. for authentication
			using plain text passwords, set the SecurityFlags
			to 0x30030)::

			  may use packet signing			0x00001
			  must use packet signing			0x01001
			  may use NTLM (most common password hash)	0x00002
			  must use NTLM					0x02002
			  may use NTLMv2				0x00004
			  must use NTLMv2				0x04004
			  may use Kerberos security			0x00008
			  must use Kerberos				0x08008
			  may use lanman (weak) password hash		0x00010
			  must use lanman password hash			0x10010
			  may use plaintext passwords			0x00020
			  must use plaintext passwords			0x20020
			  (reserved for future packet encryption)	0x00040

cifsFYI			If set to non-zero value, additional debug information
			will be logged to the system error log.  This field
			contains three flags controlling different classes of
			debugging entries.  The maximum value it can be set
			to is 7 which enables all debugging points (default 0).
			Some debugging statements are not compiled into the
			cifs kernel unless CONFIG_CIFS_DEBUG2 is enabled in the
			kernel configuration. cifsFYI may be set to one or
			nore of the following flags (7 sets them all)::

			  +-----------------------------------------------+------+
			  | log cifs informational messages		  | 0x01 |
			  +-----------------------------------------------+------+
			  | log return codes from cifs entry points	  | 0x02 |
			  +-----------------------------------------------+------+
			  | log slow responses				  | 0x04 |
			  | (ie which take longer than 1 second)	  |      |
			  |                                               |      |
			  | CONFIG_CIFS_STATS2 must be enabled in .config |      |
			  +-----------------------------------------------+------+

traceSMB		If set to one, debug information is logged to the
			system error log with the start of smb requests
			and responses (default 0)
LookupCacheEnable	If set to one, inode information is kept cached
			for one second improving performance of lookups
			(default 1)
LinuxExtensionsEnabled	If set to one then the client will attempt to
			use the CIFS "UNIX" extensions which are optional
			protocol enhancements that allow CIFS servers
			to return accurate UID/GID information as well
			as support symbolic links. If you use servers
			such as Samba that support the CIFS Unix
			extensions but do not want to use symbolic link
			support and want to map the uid and gid fields
			to values supplied at mount (rather than the
			actual values, then set this to zero. (default 1)
dfscache		List the content of the DFS cache.
			If set to 0, the client will clear the cache.
======================= =======================================================

These experimental features and tracing can be enabled by changing flags in
/proc/fs/cifs (after the cifs module has been installed or built into the
kernel, e.g.  insmod cifs).  To enable a feature set it to 1 e.g.  to enable
tracing to the kernel message log type::

	echo 7 > /proc/fs/cifs/cifsFYI

cifsFYI functions as a bit mask. Setting it to 1 enables additional kernel
logging of various informational messages.  2 enables logging of non-zero
SMB return codes while 4 enables logging of requests that take longer
than one second to complete (except for byte range lock requests).
Setting it to 4 requires CONFIG_CIFS_STATS2 to be set in kernel configuration
(.config). Setting it to seven enables all three.  Finally, tracing
the start of smb requests and responses can be enabled via::

	echo 1 > /proc/fs/cifs/traceSMB

Per share (per client mount) statistics are available in /proc/fs/cifs/Stats.
Additional information is available if CONFIG_CIFS_STATS2 is enabled in the
kernel configuration (.config).  The statistics returned include counters which
represent the number of attempted and failed (ie non-zero return code from the
server) SMB3 (or cifs) requests grouped by request type (read, write, close etc.).
Also recorded is the total bytes read and bytes written to the server for
that share.  Note that due to client caching effects this can be less than the
number of bytes read and written by the application running on the client.
Statistics can be reset to zero by ``echo 0 > /proc/fs/cifs/Stats`` which may be
useful if comparing performance of two different scenarios.

Also note that ``cat /proc/fs/cifs/DebugData`` will display information about
the active sessions and the shares that are mounted.

Enabling Kerberos (extended security) works but requires version 1.2 or later
of the helper program cifs.upcall to be present and to be configured in the
/etc/request-key.conf file.  The cifs.upcall helper program is from the Samba
project(https://www.samba.org). NTLM and NTLMv2 and LANMAN support do not
require this helper. Note that NTLMv2 security (which does not require the
cifs.upcall helper program), instead of using Kerberos, is sufficient for
some use cases.

DFS support allows transparent redirection to shares in an MS-DFS name space.
In addition, DFS support for target shares which are specified as UNC
names which begin with host names (rather than IP addresses) requires
a user space helper (such as cifs.upcall) to be present in order to
translate host names to ip address, and the user space helper must also
be configured in the file /etc/request-key.conf.  Samba, Windows servers and
many NAS appliances support DFS as a way of constructing a global name
space to ease network configuration and improve reliability.

To use cifs Kerberos and DFS support, the Linux keyutils package should be
installed and something like the following lines should be added to the
/etc/request-key.conf file::

  create cifs.spnego * * /usr/local/sbin/cifs.upcall %k
  create dns_resolver * * /usr/local/sbin/cifs.upcall %k

CIFS kernel module parameters
=============================
These module parameters can be specified or modified either during the time of
module loading or during the runtime by using the interface::

	/sys/module/cifs/parameters/<param>

i.e.::

    echo "value" > /sys/module/cifs/parameters/<param>

================= ==========================================================
1. enable_oplocks Enable or disable oplocks. Oplocks are enabled by default.
		  [Y/y/1]. To disable use any of [N/n/0].
================= ==========================================================


code:
#!/usr/bin/perl -w
#
# winucase_convert.pl -- convert "Windows 8 Upper Case Mapping Table.txt" to
#                        a two-level set of C arrays.
#
#   Copyright 2013: Jeff Layton <jlayton@redhat.com>
#
#   This program is free software: you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation, either version 3 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program.  If not, see <https://www.gnu.org/licenses/>.
#

while(<>) {
	next if (!/^0x(..)(..)\t0x(....)\t/);
	$firstchar = hex($1);
	$secondchar = hex($2);
	$uppercase = hex($3);

	$top[$firstchar][$secondchar] = $uppercase;
}

for ($i = 0; $i < 256; $i++) {
	next if (!$top[$i]);

	printf("static const wchar_t t2_%2.2x[256] = {", $i);
	for ($j = 0; $j < 256; $j++) {
		if (($j % 8) == 0) {
			print "\n\t";
		} else {
			print " ";
		}
		printf("0x%4.4x,", $top[$i][$j] ? $top[$i][$j] : 0);
	}
	print "\n};\n\n";
}

printf("static const wchar_t *const toplevel[256] = {", $i);
for ($i = 0; $i < 256; $i++) {
	if (($i % 8) == 0) {
		print "\n\t";
	} elsif ($top[$i]) {
		print " ";
	} else {
		print "  ";
	}

	if ($top[$i]) {
		printf("t2_%2.2x,", $i);
	} else {
		print "NULL,";
	}
}
print "\n};\n\n";


code:
Clearing WARN_ONCE
------------------

WARN_ONCE / WARN_ON_ONCE / printk_once only emit a message once.

echo 1 > /sys/kernel/debug/clear_warn_once

clears the state and allows the warnings to print once again.
This can be useful after test suite runs to reproduce problems.


code:
========
CPU load
========

Linux exports various bits of information via ``/proc/stat`` and
``/proc/uptime`` that userland tools, such as top(1), use to calculate
the average time system spent in a particular state, for example::

    $ iostat
    Linux 2.6.18.3-exp (linmac)     02/20/2007

    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
              10.01    0.00    2.92    5.44    0.00   81.63

    ...

Here the system thinks that over the default sampling period the
system spent 10.01% of the time doing work in user space, 2.92% in the
kernel, and was overall 81.63% of the time idle.

In most cases the ``/proc/stat``	 information reflects the reality quite
closely, however due to the nature of how/when the kernel collects
this data sometimes it can not be trusted at all.

So how is this information collected?  Whenever timer interrupt is
signalled the kernel looks what kind of task was running at this
moment and increments the counter that corresponds to this tasks
kind/state.  The problem with this is that the system could have
switched between various states multiple times between two timer
interrupts yet the counter is incremented only for the last state.

Example
-------

If we imagine the system with one task that periodically burns cycles
in the following manner::

     time line between two timer interrupts
    |--------------------------------------|
     ^                                    ^
     |_ something begins working          |
                                          |_ something goes to sleep
                                         (only to be awaken quite soon)

In the above situation the system will be 0% loaded according to the
``/proc/stat`` (since the timer interrupt will always happen when the
system is executing the idle handler), but in reality the load is
closer to 99%.

One can imagine many more situations where this behavior of the kernel
will lead to quite erratic information inside ``/proc/stat``::

	/* gcc -o hog smallhog.c */
	#include <time.h>
	#include <limits.h>
	#include <signal.h>
	#include <sys/time.h>
	#define HIST 10

	static volatile sig_atomic_t stop;

	static void sighandler(int signr)
	{
		(void) signr;
		stop = 1;
	}

	static unsigned long hog (unsigned long niters)
	{
		stop = 0;
		while (!stop && --niters);
		return niters;
	}

	int main (void)
	{
		int i;
		struct itimerval it = {
			.it_interval = { .tv_sec = 0, .tv_usec = 1 },
			.it_value    = { .tv_sec = 0, .tv_usec = 1 } };
		sigset_t set;
		unsigned long v[HIST];
		double tmp = 0.0;
		unsigned long n;
		signal(SIGALRM, &sighandler);
		setitimer(ITIMER_REAL, &it, NULL);

		hog (ULONG_MAX);
		for (i = 0; i < HIST; ++i) v[i] = ULONG_MAX - hog(ULONG_MAX);
		for (i = 0; i < HIST; ++i) tmp += v[i];
		tmp /= HIST;
		n = tmp - (tmp / 3.0);

		sigemptyset(&set);
		sigaddset(&set, SIGALRM);

		for (;;) {
			hog(n);
			sigwait(&set, &i);
		}
		return 0;
	}

References
----------

- https://lore.kernel.org/r/loom.20070212T063225-663@post.gmane.org
- Documentation/filesystems/proc.rst (1.8)

Thanks
------

Con Kolivas, Pavel Machek


code:
===========================================
How CPU topology info is exported via sysfs
===========================================

CPU topology info is exported via sysfs. Items (attributes) are similar
to /proc/cpuinfo output of some architectures. They reside in
/sys/devices/system/cpu/cpuX/topology/. Please refer to the ABI file:
Documentation/ABI/stable/sysfs-devices-system-cpu.

Architecture-neutral, drivers/base/topology.c, exports these attributes.
However the die, cluster, book, and drawer hierarchy related sysfs files will
only be created if an architecture provides the related macros as described
below.

For an architecture to support this feature, it must define some of
these macros in include/asm-XXX/topology.h::

	#define topology_physical_package_id(cpu)
	#define topology_die_id(cpu)
	#define topology_cluster_id(cpu)
	#define topology_core_id(cpu)
	#define topology_book_id(cpu)
	#define topology_drawer_id(cpu)
	#define topology_sibling_cpumask(cpu)
	#define topology_core_cpumask(cpu)
	#define topology_cluster_cpumask(cpu)
	#define topology_die_cpumask(cpu)
	#define topology_book_cpumask(cpu)
	#define topology_drawer_cpumask(cpu)

The type of ``**_id macros`` is int.
The type of ``**_cpumask macros`` is ``(const) struct cpumask *``. The latter
correspond with appropriate ``**_siblings`` sysfs attributes (except for
topology_sibling_cpumask() which corresponds with thread_siblings).

To be consistent on all architectures, include/linux/topology.h
provides default definitions for any of the above macros that are
not defined by include/asm-XXX/topology.h:

1) topology_physical_package_id: -1
2) topology_die_id: -1
3) topology_cluster_id: -1
4) topology_core_id: 0
5) topology_book_id: -1
6) topology_drawer_id: -1
7) topology_sibling_cpumask: just the given CPU
8) topology_core_cpumask: just the given CPU
9) topology_cluster_cpumask: just the given CPU
10) topology_die_cpumask: just the given CPU
11) topology_book_cpumask:  just the given CPU
12) topology_drawer_cpumask: just the given CPU

Additionally, CPU topology information is provided under
/sys/devices/system/cpu and includes these files.  The internal
source for the output is in brackets ("[]").

    =========== ==========================================================
    kernel_max: the maximum CPU index allowed by the kernel configuration.
		[NR_CPUS-1]

    offline:	CPUs that are not online because they have been
		HOTPLUGGED off or exceed the limit of CPUs allowed by the
		kernel configuration (kernel_max above).
		[~cpu_online_mask + cpus >= NR_CPUS]

    online:	CPUs that are online and being scheduled [cpu_online_mask]

    possible:	CPUs that have been allocated resources and can be
		brought online if they are present. [cpu_possible_mask]

    present:	CPUs that have been identified as being present in the
		system. [cpu_present_mask]
    =========== ==========================================================

The format for the above output is compatible with cpulist_parse()
[see <linux/cpumask.h>].  Some examples follow.

In this example, there are 64 CPUs in the system but cpus 32-63 exceed
the kernel max which is limited to 0..31 by the NR_CPUS config option
being 32.  Note also that CPUs 2 and 4-31 are not online but could be
brought online as they are both present and possible::

     kernel_max: 31
        offline: 2,4-31,32-63
         online: 0-1,3
       possible: 0-31
        present: 0-31

In this example, the NR_CPUS config option is 128, but the kernel was
started with possible_cpus=144.  There are 4 CPUs in the system and cpu2
was manually taken offline (and is the only CPU that can be brought
online.)::

     kernel_max: 127
        offline: 2,4-127,128-143
         online: 0-1,3
       possible: 0-127
        present: 0-3

See Documentation/core-api/cpu_hotplug.rst for the possible_cpus=NUM
kernel start parameter as well as more information on the various cpumasks.


code:
=========================================
Dell Remote BIOS Update driver (dell_rbu)
=========================================

Purpose
=======

Document demonstrating the use of the Dell Remote BIOS Update driver
for updating BIOS images on Dell servers and desktops.

Scope
=====

This document discusses the functionality of the rbu driver only.
It does not cover the support needed from applications to enable the BIOS to
update itself with the image downloaded in to the memory.

Overview
========

This driver works with Dell OpenManage or Dell Update Packages for updating
the BIOS on Dell servers (starting from servers sold since 1999), desktops
and notebooks (starting from those sold in 2005).

Please go to  http://support.dell.com register and you can find info on
OpenManage and Dell Update packages (DUP).

Libsmbios can also be used to update BIOS on Dell systems go to
https://linux.dell.com/libsmbios/ for details.

Dell_RBU driver supports BIOS update using the monolithic image and packetized
image methods. In case of monolithic the driver allocates a contiguous chunk
of physical pages having the BIOS image. In case of packetized the app
using the driver breaks the image in to packets of fixed sizes and the driver
would place each packet in contiguous physical memory. The driver also
maintains a link list of packets for reading them back.

If the dell_rbu driver is unloaded all the allocated memory is freed.

The rbu driver needs to have an application (as mentioned above) which will
inform the BIOS to enable the update in the next system reboot.

The user should not unload the rbu driver after downloading the BIOS image
or updating.

The driver load creates the following directories under the /sys file system::

	/sys/class/firmware/dell_rbu/loading
	/sys/class/firmware/dell_rbu/data
	/sys/devices/platform/dell_rbu/image_type
	/sys/devices/platform/dell_rbu/data
	/sys/devices/platform/dell_rbu/packet_size

The driver supports two types of update mechanism; monolithic and packetized.
These update mechanism depends upon the BIOS currently running on the system.
Most of the Dell systems support a monolithic update where the BIOS image is
copied to a single contiguous block of physical memory.

In case of packet mechanism the single memory can be broken in smaller chunks
of contiguous memory and the BIOS image is scattered in these packets.

By default the driver uses monolithic memory for the update type. This can be
changed to packets during the driver load time by specifying the load
parameter image_type=packet.  This can also be changed later as below::

	echo packet > /sys/devices/platform/dell_rbu/image_type

In packet update mode the packet size has to be given before any packets can
be downloaded. It is done as below::

	echo XXXX > /sys/devices/platform/dell_rbu/packet_size

In the packet update mechanism, the user needs to create a new file having
packets of data arranged back to back. It can be done as follows:
The user creates packets header, gets the chunk of the BIOS image and
places it next to the packetheader; now, the packetheader + BIOS image chunk
added together should match the specified packet_size. This makes one
packet, the user needs to create more such packets out of the entire BIOS
image file and then arrange all these packets back to back in to one single
file.

This file is then copied to /sys/class/firmware/dell_rbu/data.
Once this file gets to the driver, the driver extracts packet_size data from
the file and spreads it across the physical memory in contiguous packet_sized
space.

This method makes sure that all the packets get to the driver in a single operation.

In monolithic update the user simply get the BIOS image (.hdr file) and copies
to the data file as is without any change to the BIOS image itself.

Do the steps below to download the BIOS image.

1) echo 1 > /sys/class/firmware/dell_rbu/loading
2) cp bios_image.hdr /sys/class/firmware/dell_rbu/data
3) echo 0 > /sys/class/firmware/dell_rbu/loading

The /sys/class/firmware/dell_rbu/ entries will remain till the following is
done.

::

	echo -1 > /sys/class/firmware/dell_rbu/loading

Until this step is completed the driver cannot be unloaded.

Also echoing either mono, packet or init in to image_type will free up the
memory allocated by the driver.

If a user by accident executes steps 1 and 3 above without executing step 2;
it will make the /sys/class/firmware/dell_rbu/ entries disappear.

The entries can be recreated by doing the following::

	echo init > /sys/devices/platform/dell_rbu/image_type

.. note:: echoing init in image_type does not change its original value.

Also the driver provides /sys/devices/platform/dell_rbu/data readonly file to
read back the image downloaded.

.. note::

   After updating the BIOS image a user mode application needs to execute
   code which sends the BIOS update request to the BIOS. So on the next reboot
   the BIOS knows about the new image downloaded and it updates itself.
   Also don't unload the rbu driver if the image has to be updated.


code:
=============================
Guidance for writing policies
=============================

Try to keep transactionality out of it.  The core is careful to
avoid asking about anything that is migrating.  This is a pain, but
makes it easier to write the policies.

Mappings are loaded into the policy at construction time.

Every bio that is mapped by the target is referred to the policy.
The policy can return a simple HIT or MISS or issue a migration.

Currently there's no way for the policy to issue background work,
e.g. to start writing back dirty blocks that are going to be evicted
soon.

Because we map bios, rather than requests it's easy for the policy
to get fooled by many small bios.  For this reason the core target
issues periodic ticks to the policy.  It's suggested that the policy
doesn't update states (eg, hit counts) for a block more than once
for each tick.  The core ticks by watching bios complete, and so
trying to see when the io scheduler has let the ios run.

Overview of supplied cache replacement policies
===============================================

multiqueue (mq)
---------------

This policy is now an alias for smq (see below).

The following tunables are accepted, but have no effect::

	'sequential_threshold <#nr_sequential_ios>'
	'random_threshold <#nr_random_ios>'
	'read_promote_adjustment <value>'
	'write_promote_adjustment <value>'
	'discard_promote_adjustment <value>'

Stochastic multiqueue (smq)
---------------------------

This policy is the default.

The stochastic multi-queue (smq) policy addresses some of the problems
with the multiqueue (mq) policy.

The smq policy (vs mq) offers the promise of less memory utilization,
improved performance and increased adaptability in the face of changing
workloads.  smq also does not have any cumbersome tuning knobs.

Users may switch from "mq" to "smq" simply by appropriately reloading a
DM table that is using the cache target.  Doing so will cause all of the
mq policy's hints to be dropped.  Also, performance of the cache may
degrade slightly until smq recalculates the origin device's hotspots
that should be cached.

Memory usage
^^^^^^^^^^^^

The mq policy used a lot of memory; 88 bytes per cache block on a 64
bit machine.

smq uses 28bit indexes to implement its data structures rather than
pointers.  It avoids storing an explicit hit count for each block.  It
has a 'hotspot' queue, rather than a pre-cache, which uses a quarter of
the entries (each hotspot block covers a larger area than a single
cache block).

All this means smq uses ~25bytes per cache block.  Still a lot of
memory, but a substantial improvement nontheless.

Level balancing
^^^^^^^^^^^^^^^

mq placed entries in different levels of the multiqueue structures
based on their hit count (~ln(hit count)).  This meant the bottom
levels generally had the most entries, and the top ones had very
few.  Having unbalanced levels like this reduced the efficacy of the
multiqueue.

smq does not maintain a hit count, instead it swaps hit entries with
the least recently used entry from the level above.  The overall
ordering being a side effect of this stochastic process.  With this
scheme we can decide how many entries occupy each multiqueue level,
resulting in better promotion/demotion decisions.

Adaptability:
The mq policy maintained a hit count for each cache block.  For a
different block to get promoted to the cache its hit count has to
exceed the lowest currently in the cache.  This meant it could take a
long time for the cache to adapt between varying IO patterns.

smq doesn't maintain hit counts, so a lot of this problem just goes
away.  In addition it tracks performance of the hotspot queue, which
is used to decide which blocks to promote.  If the hotspot queue is
performing badly then it starts moving entries more quickly between
levels.  This lets it adapt to new IO patterns very quickly.

Performance
^^^^^^^^^^^

Testing smq shows substantially better performance than mq.

cleaner
-------

The cleaner writes back all dirty blocks in a cache to decommission it.

Examples
========

The syntax for a table is::

	cache <metadata dev> <cache dev> <origin dev> <block size>
	<#feature_args> [<feature arg>]*
	<policy> <#policy_args> [<policy arg>]*

The syntax to send a message using the dmsetup command is::

	dmsetup message <mapped device> 0 sequential_threshold 1024
	dmsetup message <mapped device> 0 random_threshold 8

Using dmsetup::

	dmsetup create blah --table "0 268435456 cache /dev/sdb /dev/sdc \
	    /dev/sdd 512 0 mq 4 sequential_threshold 1024 random_threshold 8"
	creates a 128GB large mapped device named 'blah' with the
	sequential threshold set to 1024 and the random_threshold set to 8.


code:
=====
Cache
=====

Introduction
============

dm-cache is a device mapper target written by Joe Thornber, Heinz
Mauelshagen, and Mike Snitzer.

It aims to improve performance of a block device (eg, a spindle) by
dynamically migrating some of its data to a faster, smaller device
(eg, an SSD).

This device-mapper solution allows us to insert this caching at
different levels of the dm stack, for instance above the data device for
a thin-provisioning pool.  Caching solutions that are integrated more
closely with the virtual memory system should give better performance.

The target reuses the metadata library used in the thin-provisioning
library.

The decision as to what data to migrate and when is left to a plug-in
policy module.  Several of these have been written as we experiment,
and we hope other people will contribute others for specific io
scenarios (eg. a vm image server).

Glossary
========

  Migration
	       Movement of the primary copy of a logical block from one
	       device to the other.
  Promotion
	       Migration from slow device to fast device.
  Demotion
	       Migration from fast device to slow device.

The origin device always contains a copy of the logical block, which
may be out of date or kept in sync with the copy on the cache device
(depending on policy).

Design
======

Sub-devices
-----------

The target is constructed by passing three devices to it (along with
other parameters detailed later):

1. An origin device - the big, slow one.

2. A cache device - the small, fast one.

3. A small metadata device - records which blocks are in the cache,
   which are dirty, and extra hints for use by the policy object.
   This information could be put on the cache device, but having it
   separate allows the volume manager to configure it differently,
   e.g. as a mirror for extra robustness.  This metadata device may only
   be used by a single cache device.

Fixed block size
----------------

The origin is divided up into blocks of a fixed size.  This block size
is configurable when you first create the cache.  Typically we've been
using block sizes of 256KB - 1024KB.  The block size must be between 64
sectors (32KB) and 2097152 sectors (1GB) and a multiple of 64 sectors (32KB).

Having a fixed block size simplifies the target a lot.  But it is
something of a compromise.  For instance, a small part of a block may be
getting hit a lot, yet the whole block will be promoted to the cache.
So large block sizes are bad because they waste cache space.  And small
block sizes are bad because they increase the amount of metadata (both
in core and on disk).

Cache operating modes
---------------------

The cache has three operating modes: writeback, writethrough and
passthrough.

If writeback, the default, is selected then a write to a block that is
cached will go only to the cache and the block will be marked dirty in
the metadata.

If writethrough is selected then a write to a cached block will not
complete until it has hit both the origin and cache devices.  Clean
blocks should remain clean.

If passthrough is selected, useful when the cache contents are not known
to be coherent with the origin device, then all reads are served from
the origin device (all reads miss the cache) and all writes are
forwarded to the origin device; additionally, write hits cause cache
block invalidates.  To enable passthrough mode the cache must be clean.
Passthrough mode allows a cache device to be activated without having to
worry about coherency.  Coherency that exists is maintained, although
the cache will gradually cool as writes take place.  If the coherency of
the cache can later be verified, or established through use of the
"invalidate_cblocks" message, the cache device can be transitioned to
writethrough or writeback mode while still warm.  Otherwise, the cache
contents can be discarded prior to transitioning to the desired
operating mode.

A simple cleaner policy is provided, which will clean (write back) all
dirty blocks in a cache.  Useful for decommissioning a cache or when
shrinking a cache.  Shrinking the cache's fast device requires all cache
blocks, in the area of the cache being removed, to be clean.  If the
area being removed from the cache still contains dirty blocks the resize
will fail.  Care must be taken to never reduce the volume used for the
cache's fast device until the cache is clean.  This is of particular
importance if writeback mode is used.  Writethrough and passthrough
modes already maintain a clean cache.  Future support to partially clean
the cache, above a specified threshold, will allow for keeping the cache
warm and in writeback mode during resize.

Migration throttling
--------------------

Migrating data between the origin and cache device uses bandwidth.
The user can set a throttle to prevent more than a certain amount of
migration occurring at any one time.  Currently we're not taking any
account of normal io traffic going to the devices.  More work needs
doing here to avoid migrating during those peak io moments.

For the time being, a message "migration_threshold <#sectors>"
can be used to set the maximum number of sectors being migrated,
the default being 2048 sectors (1MB).

Updating on-disk metadata
-------------------------

On-disk metadata is committed every time a FLUSH or FUA bio is written.
If no such requests are made then commits will occur every second.  This
means the cache behaves like a physical disk that has a volatile write
cache.  If power is lost you may lose some recent writes.  The metadata
should always be consistent in spite of any crash.

The 'dirty' state for a cache block changes far too frequently for us
to keep updating it on the fly.  So we treat it as a hint.  In normal
operation it will be written when the dm device is suspended.  If the
system crashes all cache blocks will be assumed dirty when restarted.

Per-block policy hints
----------------------

Policy plug-ins can store a chunk of data per cache block.  It's up to
the policy how big this chunk is, but it should be kept small.  Like the
dirty flags this data is lost if there's a crash so a safe fallback
value should always be possible.

Policy hints affect performance, not correctness.

Policy messaging
----------------

Policies will have different tunables, specific to each one, so we
need a generic way of getting and setting these.  Device-mapper
messages are used.  Refer to cache-policies.txt.

Discard bitset resolution
-------------------------

We can avoid copying data during migration if we know the block has
been discarded.  A prime example of this is when mkfs discards the
whole block device.  We store a bitset tracking the discard state of
blocks.  However, we allow this bitset to have a different block size
from the cache blocks.  This is because we need to track the discard
state for all of the origin device (compare with the dirty bitset
which is just for the smaller cache device).

Target interface
================

Constructor
-----------

  ::

   cache <metadata dev> <cache dev> <origin dev> <block size>
         <#feature args> [<feature arg>]*
         <policy> <#policy args> [policy args]*

 ================ =======================================================
 metadata dev     fast device holding the persistent metadata
 cache dev	  fast device holding cached data blocks
 origin dev	  slow device holding original data blocks
 block size       cache unit size in sectors

 #feature args    number of feature arguments passed
 feature args     writethrough or passthrough (The default is writeback.)

 policy           the replacement policy to use
 #policy args     an even number of arguments corresponding to
                  key/value pairs passed to the policy
 policy args      key/value pairs passed to the policy
		  E.g. 'sequential_threshold 1024'
		  See cache-policies.txt for details.
 ================ =======================================================

Optional feature arguments are:

   ==================== ========================================================
   writethrough		write through caching that prohibits cache block
			content from being different from origin block content.
			Without this argument, the default behaviour is to write
			back cache block contents later for performance reasons,
			so they may differ from the corresponding origin blocks.

   passthrough		a degraded mode useful for various cache coherency
			situations (e.g., rolling back snapshots of
			underlying storage).	 Reads and writes always go to
			the origin.	If a write goes to a cached origin
			block, then the cache block is invalidated.
			To enable passthrough mode the cache must be clean.

   metadata2		use version 2 of the metadata.  This stores the dirty
			bits in a separate btree, which improves speed of
			shutting down the cache.

   no_discard_passdown	disable passing down discards from the cache
			to the origin's data device.
   ==================== ========================================================

A policy called 'default' is always registered.  This is an alias for
the policy we currently think is giving best all round performance.

As the default policy could vary between kernels, if you are relying on
the characteristics of a specific policy, always request it by name.

Status
------

::

  <metadata block size> <#used metadata blocks>/<#total metadata blocks>
  <cache block size> <#used cache blocks>/<#total cache blocks>
  <#read hits> <#read misses> <#write hits> <#write misses>
  <#demotions> <#promotions> <#dirty> <#features> <features>*
  <#core args> <core args>* <policy name> <#policy args> <policy args>*
  <cache metadata mode>

========================= =====================================================
metadata block size	  Fixed block size for each metadata block in
			  sectors
#used metadata blocks	  Number of metadata blocks used
#total metadata blocks	  Total number of metadata blocks
cache block size	  Configurable block size for the cache device
			  in sectors
#used cache blocks	  Number of blocks resident in the cache
#total cache blocks	  Total number of cache blocks
#read hits		  Number of times a READ bio has been mapped
			  to the cache
#read misses		  Number of times a READ bio has been mapped
			  to the origin
#write hits		  Number of times a WRITE bio has been mapped
			  to the cache
#write misses		  Number of times a WRITE bio has been
			  mapped to the origin
#demotions		  Number of times a block has been removed
			  from the cache
#promotions		  Number of times a block has been moved to
			  the cache
#dirty			  Number of blocks in the cache that differ
			  from the origin
#feature args		  Number of feature args to follow
feature args		  'writethrough' (optional)
#core args		  Number of core arguments (must be even)
core args		  Key/value pairs for tuning the core
			  e.g. migration_threshold
policy name		  Name of the policy
#policy args		  Number of policy arguments to follow (must be even)
policy args		  Key/value pairs e.g. sequential_threshold
cache metadata mode       ro if read-only, rw if read-write

			  In serious cases where even a read-only mode is
			  deemed unsafe no further I/O will be permitted and
			  the status will just contain the string 'Fail'.
			  The userspace recovery tools should then be used.
needs_check		  'needs_check' if set, '-' if not set
			  A metadata operation has failed, resulting in the
			  needs_check flag being set in the metadata's
			  superblock.  The metadata device must be
			  deactivated and checked/repaired before the
			  cache can be made fully operational again.
			  '-' indicates	needs_check is not set.
========================= =====================================================

Messages
--------

Policies will have different tunables, specific to each one, so we
need a generic way of getting and setting these.  Device-mapper
messages are used.  (A sysfs interface would also be possible.)

The message format is::

   <key> <value>

E.g.::

   dmsetup message my_cache 0 sequential_threshold 1024

Invalidation is removing an entry from the cache without writing it
back.  Cache blocks can be invalidated via the invalidate_cblocks
message, which takes an arbitrary number of cblock ranges.  Each cblock
range's end value is "one past the end", meaning 5-10 expresses a range
of values from 5 to 9.  Each cblock must be expressed as a decimal
value, in the future a variant message that takes cblock ranges
expressed in hexadecimal may be needed to better support efficient
invalidation of larger caches.  The cache must be in passthrough mode
when invalidate_cblocks is used::

   invalidate_cblocks [<cblock>|<cblock begin>-<cblock end>]*

E.g.::

   dmsetup message my_cache 0 invalidate_cblocks 2345 3456-4567 5678-6789

Examples
========

The test suite can be found here:

https://github.com/jthornber/device-mapper-test-suite

::

  dmsetup create my_cache --table '0 41943040 cache /dev/mapper/metadata \
	  /dev/mapper/ssd /dev/mapper/origin 512 1 writeback default 0'
  dmsetup create my_cache --table '0 41943040 cache /dev/mapper/metadata \
	  /dev/mapper/ssd /dev/mapper/origin 1024 1 writeback \
	  mq 4 sequential_threshold 1024 random_threshold 8'


code:
========
dm-delay
========

Device-Mapper's "delay" target delays reads and/or writes
and maps them to different devices.

Parameters::

    <device> <offset> <delay> [<write_device> <write_offset> <write_delay>
			       [<flush_device> <flush_offset> <flush_delay>]]

With separate write parameters, the first set is only used for reads.
Offsets are specified in sectors.
Delays are specified in milliseconds.

Example scripts
===============

::

	#!/bin/sh
	# Create device delaying rw operation for 500ms
	echo "0 `blockdev --getsz $1` delay $1 0 500" | dmsetup create delayed

::

	#!/bin/sh
	# Create device delaying only write operation for 500ms and
	# splitting reads and writes to different devices $1 $2
	echo "0 `blockdev --getsz $1` delay $1 0 0 $2 0 500" | dmsetup create delayed


code:
.. SPDX-License-Identifier: GPL-2.0-only

========
dm-clone
========

Introduction
============

dm-clone is a device mapper target which produces a one-to-one copy of an
existing, read-only source device into a writable destination device: It
presents a virtual block device which makes all data appear immediately, and
redirects reads and writes accordingly.

The main use case of dm-clone is to clone a potentially remote, high-latency,
read-only, archival-type block device into a writable, fast, primary-type device
for fast, low-latency I/O. The cloned device is visible/mountable immediately
and the copy of the source device to the destination device happens in the
background, in parallel with user I/O.

For example, one could restore an application backup from a read-only copy,
accessible through a network storage protocol (NBD, Fibre Channel, iSCSI, AoE,
etc.), into a local SSD or NVMe device, and start using the device immediately,
without waiting for the restore to complete.

When the cloning completes, the dm-clone table can be removed altogether and be
replaced, e.g., by a linear table, mapping directly to the destination device.

The dm-clone target reuses the metadata library used by the thin-provisioning
target.

Glossary
========

   Hydration
     The process of filling a region of the destination device with data from
     the same region of the source device, i.e., copying the region from the
     source to the destination device.

Once a region gets hydrated we redirect all I/O regarding it to the destination
device.

Design
======

Sub-devices
-----------

The target is constructed by passing three devices to it (along with other
parameters detailed later):

1. A source device - the read-only device that gets cloned and source of the
   hydration.

2. A destination device - the destination of the hydration, which will become a
   clone of the source device.

3. A small metadata device - it records which regions are already valid in the
   destination device, i.e., which regions have already been hydrated, or have
   been written to directly, via user I/O.

The size of the destination device must be at least equal to the size of the
source device.

Regions
-------

dm-clone divides the source and destination devices in fixed sized regions.
Regions are the unit of hydration, i.e., the minimum amount of data copied from
the source to the destination device.

The region size is configurable when you first create the dm-clone device. The
recommended region size is the same as the file system block size, which usually
is 4KB. The region size must be between 8 sectors (4KB) and 2097152 sectors
(1GB) and a power of two.

Reads and writes from/to hydrated regions are serviced from the destination
device.

A read to a not yet hydrated region is serviced directly from the source device.

A write to a not yet hydrated region will be delayed until the corresponding
region has been hydrated and the hydration of the region starts immediately.

Note that a write request with size equal to region size will skip copying of
the corresponding region from the source device and overwrite the region of the
destination device directly.

Discards
--------

dm-clone interprets a discard request to a range that hasn't been hydrated yet
as a hint to skip hydration of the regions covered by the request, i.e., it
skips copying the region's data from the source to the destination device, and
only updates its metadata.

If the destination device supports discards, then by default dm-clone will pass
down discard requests to it.

Background Hydration
--------------------

dm-clone copies continuously from the source to the destination device, until
all of the device has been copied.

Copying data from the source to the destination device uses bandwidth. The user
can set a throttle to prevent more than a certain amount of copying occurring at
any one time. Moreover, dm-clone takes into account user I/O traffic going to
the devices and pauses the background hydration when there is I/O in-flight.

A message `hydration_threshold <#regions>` can be used to set the maximum number
of regions being copied, the default being 1 region.

dm-clone employs dm-kcopyd for copying portions of the source device to the
destination device. By default, we issue copy requests of size equal to the
region size. A message `hydration_batch_size <#regions>` can be used to tune the
size of these copy requests. Increasing the hydration batch size results in
dm-clone trying to batch together contiguous regions, so we copy the data in
batches of this many regions.

When the hydration of the destination device finishes, a dm event will be sent
to user space.

Updating on-disk metadata
-------------------------

On-disk metadata is committed every time a FLUSH or FUA bio is written. If no
such requests are made then commits will occur every second. This means the
dm-clone device behaves like a physical disk that has a volatile write cache. If
power is lost you may lose some recent writes. The metadata should always be
consistent in spite of any crash.

Target Interface
================

Constructor
-----------

  ::

   clone <metadata dev> <destination dev> <source dev> <region size>
         [<#feature args> [<feature arg>]* [<#core args> [<core arg>]*]]

 ================ ==============================================================
 metadata dev     Fast device holding the persistent metadata
 destination dev  The destination device, where the source will be cloned
 source dev       Read only device containing the data that gets cloned
 region size      The size of a region in sectors

 #feature args    Number of feature arguments passed
 feature args     no_hydration or no_discard_passdown

 #core args       An even number of arguments corresponding to key/value pairs
                  passed to dm-clone
 core args        Key/value pairs passed to dm-clone, e.g. `hydration_threshold
                  256`
 ================ ==============================================================

Optional feature arguments are:

 ==================== =========================================================
 no_hydration         Create a dm-clone instance with background hydration
                      disabled
 no_discard_passdown  Disable passing down discards to the destination device
 ==================== =========================================================

Optional core arguments are:

 ================================ ==============================================
 hydration_threshold <#regions>   Maximum number of regions being copied from
                                  the source to the destination device at any
                                  one time, during background hydration.
 hydration_batch_size <#regions>  During background hydration, try to batch
                                  together contiguous regions, so we copy data
                                  from the source to the destination device in
                                  batches of this many regions.
 ================================ ==============================================

Status
------

  ::

   <metadata block size> <#used metadata blocks>/<#total metadata blocks>
   <region size> <#hydrated regions>/<#total regions> <#hydrating regions>
   <#feature args> <feature args>* <#core args> <core args>*
   <clone metadata mode>

 ======================= =======================================================
 metadata block size     Fixed block size for each metadata block in sectors
 #used metadata blocks   Number of metadata blocks used
 #total metadata blocks  Total number of metadata blocks
 region size             Configurable region size for the device in sectors
 #hydrated regions       Number of regions that have finished hydrating
 #total regions          Total number of regions to hydrate
 #hydrating regions      Number of regions currently hydrating
 #feature args           Number of feature arguments to follow
 feature args            Feature arguments, e.g. `no_hydration`
 #core args              Even number of core arguments to follow
 core args               Key/value pairs for tuning the core, e.g.
                         `hydration_threshold 256`
 clone metadata mode     ro if read-only, rw if read-write

                         In serious cases where even a read-only mode is deemed
                         unsafe no further I/O will be permitted and the status
                         will just contain the string 'Fail'. If the metadata
                         mode changes, a dm event will be sent to user space.
 ======================= =======================================================

Messages
--------

  `disable_hydration`
      Disable the background hydration of the destination device.

  `enable_hydration`
      Enable the background hydration of the destination device.

  `hydration_threshold <#regions>`
      Set background hydration threshold.

  `hydration_batch_size <#regions>`
      Set background hydration batch size.

Examples
========

Clone a device containing a file system
---------------------------------------

1. Create the dm-clone device.

   ::

    dmsetup create clone --table "0 1048576000 clone $metadata_dev $dest_dev \
      $source_dev 8 1 no_hydration"

2. Mount the device and trim the file system. dm-clone interprets the discards
   sent by the file system and it will not hydrate the unused space.

   ::

    mount /dev/mapper/clone /mnt/cloned-fs
    fstrim /mnt/cloned-fs

3. Enable background hydration of the destination device.

   ::

    dmsetup message clone 0 enable_hydration

4. When the hydration finishes, we can replace the dm-clone table with a linear
   table.

   ::

    dmsetup suspend clone
    dmsetup load clone --table "0 1048576000 linear $dest_dev 0"
    dmsetup resume clone

   The metadata device is no longer needed and can be safely discarded or reused
   for other purposes.

Known issues
============

1. We redirect reads, to not-yet-hydrated regions, to the source device. If
   reading the source device has high latency and the user repeatedly reads from
   the same regions, this behaviour could degrade performance. We should use
   these reads as hints to hydrate the relevant regions sooner. Currently, we
   rely on the page cache to cache these regions, so we hopefully don't end up
   reading them multiple times from the source device.

2. Release in-core resources, i.e., the bitmaps tracking which regions are
   hydrated, after the hydration has finished.

3. During background hydration, if we fail to read the source or write to the
   destination device, we print an error message, but the hydration process
   continues indefinitely, until it succeeds. We should stop the background
   hydration after a number of failures and emit a dm event for user space to
   notice.

Why not...?
===========

We explored the following alternatives before implementing dm-clone:

1. Use dm-cache with cache size equal to the source device and implement a new
   cloning policy:

   * The resulting cache device is not a one-to-one mirror of the source device
     and thus we cannot remove the cache device once cloning completes.

   * dm-cache writes to the source device, which violates our requirement that
     the source device must be treated as read-only.

   * Caching is semantically different from cloning.

2. Use dm-snapshot with a COW device equal to the source device:

   * dm-snapshot stores its metadata in the COW device, so the resulting device
     is not a one-to-one mirror of the source device.

   * No background copying mechanism.

   * dm-snapshot needs to commit its metadata whenever a pending exception
     completes, to ensure snapshot consistency. In the case of cloning, we don't
     need to be so strict and can rely on committing metadata every time a FLUSH
     or FUA bio is written, or periodically, like dm-thin and dm-cache do. This
     improves the performance significantly.

3. Use dm-mirror: The mirror target has a background copying/mirroring
   mechanism, but it writes to all mirrors, thus violating our requirement that
   the source device must be treated as read-only.

4. Use dm-thin's external snapshot functionality. This approach is the most
   promising among all alternatives, as the thinly-provisioned volume is a
   one-to-one mirror of the source device and handles reads and writes to
   un-provisioned/not-yet-cloned areas the same way as dm-clone does.

   Still:

   * There is no background copying mechanism, though one could be implemented.

   * Most importantly, we want to support arbitrary block devices as the
     destination of the cloning process and not restrict ourselves to
     thinly-provisioned volumes. Thin-provisioning has an inherent metadata
     overhead, for maintaining the thin volume mappings, which significantly
     degrades performance.

   Moreover, cloning a device shouldn't force the use of thin-provisioning. On
   the other hand, if we wish to use thin provisioning, we can just use a thin
   LV as dm-clone's destination device.