code:
.. SPDX-License-Identifier: GPL-2.0

.. _fsverity:

=======================================================
fs-verity: read-only file-based authenticity protection
=======================================================

Introduction
============

fs-verity (``fs/verity/``) is a support layer that filesystems can
hook into to support transparent integrity and authenticity protection
of read-only files.  Currently, it is supported by the ext4, f2fs, and
btrfs filesystems.  Like fscrypt, not too much filesystem-specific
code is needed to support fs-verity.

fs-verity is similar to `dm-verity
<https://www.kernel.org/doc/Documentation/device-mapper/verity.txt>`_
but works on files rather than block devices.  On regular files on
filesystems supporting fs-verity, userspace can execute an ioctl that
causes the filesystem to build a Merkle tree for the file and persist
it to a filesystem-specific location associated with the file.

After this, the file is made readonly, and all reads from the file are
automatically verified against the file's Merkle tree.  Reads of any
corrupted data, including mmap reads, will fail.

Userspace can use another ioctl to retrieve the root hash (actually
the "fs-verity file digest", which is a hash that includes the Merkle
tree root hash) that fs-verity is enforcing for the file.  This ioctl
executes in constant time, regardless of the file size.

fs-verity is essentially a way to hash a file in constant time,
subject to the caveat that reads which would violate the hash will
fail at runtime.

Use cases
=========

By itself, the base fs-verity feature only provides integrity
protection, i.e. detection of accidental (non-malicious) corruption.

However, because fs-verity makes retrieving the file hash extremely
efficient, it's primarily meant to be used as a tool to support
authentication (detection of malicious modifications) or auditing
(logging file hashes before use).

Trusted userspace code (e.g. operating system code running on a
read-only partition that is itself authenticated by dm-verity) can
authenticate the contents of an fs-verity file by using the
`FS_IOC_MEASURE_VERITY`_ ioctl to retrieve its hash, then verifying a
digital signature of it.

A standard file hash could be used instead of fs-verity.  However,
this is inefficient if the file is large and only a small portion may
be accessed.  This is often the case for Android application package
(APK) files, for example.  These typically contain many translations,
classes, and other resources that are infrequently or even never
accessed on a particular device.  It would be slow and wasteful to
read and hash the entire file before starting the application.

Unlike an ahead-of-time hash, fs-verity also re-verifies data each
time it's paged in.  This ensures that malicious disk firmware can't
undetectably change the contents of the file at runtime.

fs-verity does not replace or obsolete dm-verity.  dm-verity should
still be used on read-only filesystems.  fs-verity is for files that
must live on a read-write filesystem because they are independently
updated and potentially user-installed, so dm-verity cannot be used.

The base fs-verity feature is a hashing mechanism only; actually
authenticating the files may be done by:

* Userspace-only

* Builtin signature verification + userspace policy

  fs-verity optionally supports a simple signature verification
  mechanism where users can configure the kernel to require that
  all fs-verity files be signed by a key loaded into a keyring;
  see `Built-in signature verification`_.

* Integrity Measurement Architecture (IMA)

  IMA supports including fs-verity file digests and signatures in the
  IMA measurement list and verifying fs-verity based file signatures
  stored as security.ima xattrs, based on policy.

User API
========

FS_IOC_ENABLE_VERITY
--------------------

The FS_IOC_ENABLE_VERITY ioctl enables fs-verity on a file.  It takes
in a pointer to a struct fsverity_enable_arg, defined as
follows::

    struct fsverity_enable_arg {
            __u32 version;
            __u32 hash_algorithm;
            __u32 block_size;
            __u32 salt_size;
            __u64 salt_ptr;
            __u32 sig_size;
            __u32 __reserved1;
            __u64 sig_ptr;
            __u64 __reserved2[11];
    };

This structure contains the parameters of the Merkle tree to build for
the file, and optionally contains a signature.  It must be initialized
as follows:

- ``version`` must be 1.
- ``hash_algorithm`` must be the identifier for the hash algorithm to
  use for the Merkle tree, such as FS_VERITY_HASH_ALG_SHA256.  See
  ``include/uapi/linux/fsverity.h`` for the list of possible values.
- ``block_size`` must be the Merkle tree block size.  Currently, this
  must be equal to the system page size, which is usually 4096 bytes.
  Other sizes may be supported in the future.  This value is not
  necessarily the same as the filesystem block size.
- ``salt_size`` is the size of the salt in bytes, or 0 if no salt is
  provided.  The salt is a value that is prepended to every hashed
  block; it can be used to personalize the hashing for a particular
  file or device.  Currently the maximum salt size is 32 bytes.
- ``salt_ptr`` is the pointer to the salt, or NULL if no salt is
  provided.
- ``sig_size`` is the size of the signature in bytes, or 0 if no
  signature is provided.  Currently the signature is (somewhat
  arbitrarily) limited to 16128 bytes.  See `Built-in signature
  verification`_ for more information.
- ``sig_ptr``  is the pointer to the signature, or NULL if no
  signature is provided.
- All reserved fields must be zeroed.

FS_IOC_ENABLE_VERITY causes the filesystem to build a Merkle tree for
the file and persist it to a filesystem-specific location associated
with the file, then mark the file as a verity file.  This ioctl may
take a long time to execute on large files, and it is interruptible by
fatal signals.

FS_IOC_ENABLE_VERITY checks for write access to the inode.  However,
it must be executed on an O_RDONLY file descriptor and no processes
can have the file open for writing.  Attempts to open the file for
writing while this ioctl is executing will fail with ETXTBSY.  (This
is necessary to guarantee that no writable file descriptors will exist
after verity is enabled, and to guarantee that the file's contents are
stable while the Merkle tree is being built over it.)

On success, FS_IOC_ENABLE_VERITY returns 0, and the file becomes a
verity file.  On failure (including the case of interruption by a
fatal signal), no changes are made to the file.

FS_IOC_ENABLE_VERITY can fail with the following errors:

- ``EACCES``: the process does not have write access to the file
- ``EBADMSG``: the signature is malformed
- ``EBUSY``: this ioctl is already running on the file
- ``EEXIST``: the file already has verity enabled
- ``EFAULT``: the caller provided inaccessible memory
- ``EINTR``: the operation was interrupted by a fatal signal
- ``EINVAL``: unsupported version, hash algorithm, or block size; or
  reserved bits are set; or the file descriptor refers to neither a
  regular file nor a directory.
- ``EISDIR``: the file descriptor refers to a directory
- ``EKEYREJECTED``: the signature doesn't match the file
- ``EMSGSIZE``: the salt or signature is too long
- ``ENOKEY``: the fs-verity keyring doesn't contain the certificate
  needed to verify the signature
- ``ENOPKG``: fs-verity recognizes the hash algorithm, but it's not
  available in the kernel's crypto API as currently configured (e.g.
  for SHA-512, missing CONFIG_CRYPTO_SHA512).
- ``ENOTTY``: this type of filesystem does not implement fs-verity
- ``EOPNOTSUPP``: the kernel was not configured with fs-verity
  support; or the filesystem superblock has not had the 'verity'
  feature enabled on it; or the filesystem does not support fs-verity
  on this file.  (See `Filesystem support`_.)
- ``EPERM``: the file is append-only; or, a signature is required and
  one was not provided.
- ``EROFS``: the filesystem is read-only
- ``ETXTBSY``: someone has the file open for writing.  This can be the
  caller's file descriptor, another open file descriptor, or the file
  reference held by a writable memory map.

FS_IOC_MEASURE_VERITY
---------------------

The FS_IOC_MEASURE_VERITY ioctl retrieves the digest of a verity file.
The fs-verity file digest is a cryptographic digest that identifies
the file contents that are being enforced on reads; it is computed via
a Merkle tree and is different from a traditional full-file digest.

This ioctl takes in a pointer to a variable-length structure::

    struct fsverity_digest {
            __u16 digest_algorithm;
            __u16 digest_size; /* input/output */
            __u8 digest[];
    };

``digest_size`` is an input/output field.  On input, it must be
initialized to the number of bytes allocated for the variable-length
``digest`` field.

On success, 0 is returned and the kernel fills in the structure as
follows:

- ``digest_algorithm`` will be the hash algorithm used for the file
  digest.  It will match ``fsverity_enable_arg::hash_algorithm``.
- ``digest_size`` will be the size of the digest in bytes, e.g. 32
  for SHA-256.  (This can be redundant with ``digest_algorithm``.)
- ``digest`` will be the actual bytes of the digest.

FS_IOC_MEASURE_VERITY is guaranteed to execute in constant time,
regardless of the size of the file.

FS_IOC_MEASURE_VERITY can fail with the following errors:

- ``EFAULT``: the caller provided inaccessible memory
- ``ENODATA``: the file is not a verity file
- ``ENOTTY``: this type of filesystem does not implement fs-verity
- ``EOPNOTSUPP``: the kernel was not configured with fs-verity
  support, or the filesystem superblock has not had the 'verity'
  feature enabled on it.  (See `Filesystem support`_.)
- ``EOVERFLOW``: the digest is longer than the specified
  ``digest_size`` bytes.  Try providing a larger buffer.

FS_IOC_READ_VERITY_METADATA
---------------------------

The FS_IOC_READ_VERITY_METADATA ioctl reads verity metadata from a
verity file.  This ioctl is available since Linux v5.12.

This ioctl allows writing a server program that takes a verity file
and serves it to a client program, such that the client can do its own
fs-verity compatible verification of the file.  This only makes sense
if the client doesn't trust the server and if the server needs to
provide the storage for the client.

This is a fairly specialized use case, and most fs-verity users won't
need this ioctl.

This ioctl takes in a pointer to the following structure::

   #define FS_VERITY_METADATA_TYPE_MERKLE_TREE     1
   #define FS_VERITY_METADATA_TYPE_DESCRIPTOR      2
   #define FS_VERITY_METADATA_TYPE_SIGNATURE       3

   struct fsverity_read_metadata_arg {
           __u64 metadata_type;
           __u64 offset;
           __u64 length;
           __u64 buf_ptr;
           __u64 __reserved;
   };

``metadata_type`` specifies the type of metadata to read:

- ``FS_VERITY_METADATA_TYPE_MERKLE_TREE`` reads the blocks of the
  Merkle tree.  The blocks are returned in order from the root level
  to the leaf level.  Within each level, the blocks are returned in
  the same order that their hashes are themselves hashed.
  See `Merkle tree`_ for more information.

- ``FS_VERITY_METADATA_TYPE_DESCRIPTOR`` reads the fs-verity
  descriptor.  See `fs-verity descriptor`_.

- ``FS_VERITY_METADATA_TYPE_SIGNATURE`` reads the signature which was
  passed to FS_IOC_ENABLE_VERITY, if any.  See `Built-in signature
  verification`_.

The semantics are similar to those of ``pread()``.  ``offset``
specifies the offset in bytes into the metadata item to read from, and
``length`` specifies the maximum number of bytes to read from the
metadata item.  ``buf_ptr`` is the pointer to the buffer to read into,
cast to a 64-bit integer.  ``__reserved`` must be 0.  On success, the
number of bytes read is returned.  0 is returned at the end of the
metadata item.  The returned length may be less than ``length``, for
example if the ioctl is interrupted.

The metadata returned by FS_IOC_READ_VERITY_METADATA isn't guaranteed
to be authenticated against the file digest that would be returned by
`FS_IOC_MEASURE_VERITY`_, as the metadata is expected to be used to
implement fs-verity compatible verification anyway (though absent a
malicious disk, the metadata will indeed match).  E.g. to implement
this ioctl, the filesystem is allowed to just read the Merkle tree
blocks from disk without actually verifying the path to the root node.

FS_IOC_READ_VERITY_METADATA can fail with the following errors:

- ``EFAULT``: the caller provided inaccessible memory
- ``EINTR``: the ioctl was interrupted before any data was read
- ``EINVAL``: reserved fields were set, or ``offset + length``
  overflowed
- ``ENODATA``: the file is not a verity file, or
  FS_VERITY_METADATA_TYPE_SIGNATURE was requested but the file doesn't
  have a built-in signature
- ``ENOTTY``: this type of filesystem does not implement fs-verity, or
  this ioctl is not yet implemented on it
- ``EOPNOTSUPP``: the kernel was not configured with fs-verity
  support, or the filesystem superblock has not had the 'verity'
  feature enabled on it.  (See `Filesystem support`_.)

FS_IOC_GETFLAGS
---------------

The existing ioctl FS_IOC_GETFLAGS (which isn't specific to fs-verity)
can also be used to check whether a file has fs-verity enabled or not.
To do so, check for FS_VERITY_FL (0x00100000) in the returned flags.

The verity flag is not settable via FS_IOC_SETFLAGS.  You must use
FS_IOC_ENABLE_VERITY instead, since parameters must be provided.

statx
-----

Since Linux v5.5, the statx() system call sets STATX_ATTR_VERITY if
the file has fs-verity enabled.  This can perform better than
FS_IOC_GETFLAGS and FS_IOC_MEASURE_VERITY because it doesn't require
opening the file, and opening verity files can be expensive.

Accessing verity files
======================

Applications can transparently access a verity file just like a
non-verity one, with the following exceptions:

- Verity files are readonly.  They cannot be opened for writing or
  truncate()d, even if the file mode bits allow it.  Attempts to do
  one of these things will fail with EPERM.  However, changes to
  metadata such as owner, mode, timestamps, and xattrs are still
  allowed, since these are not measured by fs-verity.  Verity files
  can also still be renamed, deleted, and linked to.

- Direct I/O is not supported on verity files.  Attempts to use direct
  I/O on such files will fall back to buffered I/O.

- DAX (Direct Access) is not supported on verity files, because this
  would circumvent the data verification.

- Reads of data that doesn't match the verity Merkle tree will fail
  with EIO (for read()) or SIGBUS (for mmap() reads).

- If the sysctl "fs.verity.require_signatures" is set to 1 and the
  file is not signed by a key in the fs-verity keyring, then opening
  the file will fail.  See `Built-in signature verification`_.

Direct access to the Merkle tree is not supported.  Therefore, if a
verity file is copied, or is backed up and restored, then it will lose
its "verity"-ness.  fs-verity is primarily meant for files like
executables that are managed by a package manager.

File digest computation
=======================

This section describes how fs-verity hashes the file contents using a
Merkle tree to produce the digest which cryptographically identifies
the file contents.  This algorithm is the same for all filesystems
that support fs-verity.

Userspace only needs to be aware of this algorithm if it needs to
compute fs-verity file digests itself, e.g. in order to sign files.

.. _fsverity_merkle_tree:

Merkle tree
-----------

The file contents is divided into blocks, where the block size is
configurable but is usually 4096 bytes.  The end of the last block is
zero-padded if needed.  Each block is then hashed, producing the first
level of hashes.  Then, the hashes in this first level are grouped
into 'blocksize'-byte blocks (zero-padding the ends as needed) and
these blocks are hashed, producing the second level of hashes.  This
proceeds up the tree until only a single block remains.  The hash of
this block is the "Merkle tree root hash".

If the file fits in one block and is nonempty, then the "Merkle tree
root hash" is simply the hash of the single data block.  If the file
is empty, then the "Merkle tree root hash" is all zeroes.

The "blocks" here are not necessarily the same as "filesystem blocks".

If a salt was specified, then it's zero-padded to the closest multiple
of the input size of the hash algorithm's compression function, e.g.
64 bytes for SHA-256 or 128 bytes for SHA-512.  The padded salt is
prepended to every data or Merkle tree block that is hashed.

The purpose of the block padding is to cause every hash to be taken
over the same amount of data, which simplifies the implementation and
keeps open more possibilities for hardware acceleration.  The purpose
of the salt padding is to make the salting "free" when the salted hash
state is precomputed, then imported for each hash.

Example: in the recommended configuration of SHA-256 and 4K blocks,
128 hash values fit in each block.  Thus, each level of the Merkle
tree is approximately 128 times smaller than the previous, and for
large files the Merkle tree's size converges to approximately 1/127 of
the original file size.  However, for small files, the padding is
significant, making the space overhead proportionally more.

.. _fsverity_descriptor:

fs-verity descriptor
--------------------

By itself, the Merkle tree root hash is ambiguous.  For example, it
can't a distinguish a large file from a small second file whose data
is exactly the top-level hash block of the first file.  Ambiguities
also arise from the convention of padding to the next block boundary.

To solve this problem, the fs-verity file digest is actually computed
as a hash of the following structure, which contains the Merkle tree
root hash as well as other fields such as the file size::

    struct fsverity_descriptor {
            __u8 version;           /* must be 1 */
            __u8 hash_algorithm;    /* Merkle tree hash algorithm */
            __u8 log_blocksize;     /* log2 of size of data and tree blocks */
            __u8 salt_size;         /* size of salt in bytes; 0 if none */
            __le32 __reserved_0x04; /* must be 0 */
            __le64 data_size;       /* size of file the Merkle tree is built over */
            __u8 root_hash[64];     /* Merkle tree root hash */
            __u8 salt[32];          /* salt prepended to each hashed block */
            __u8 __reserved[144];   /* must be 0's */
    };

Built-in signature verification
===============================

With CONFIG_FS_VERITY_BUILTIN_SIGNATURES=y, fs-verity supports putting
a portion of an authentication policy (see `Use cases`_) in the
kernel.  Specifically, it adds support for:

1. At fs-verity module initialization time, a keyring ".fs-verity" is
   created.  The root user can add trusted X.509 certificates to this
   keyring using the add_key() system call, then (when done)
   optionally use keyctl_restrict_keyring() to prevent additional
   certificates from being added.

2. `FS_IOC_ENABLE_VERITY`_ accepts a pointer to a PKCS#7 formatted
   detached signature in DER format of the file's fs-verity digest.
   On success, this signature is persisted alongside the Merkle tree.
   Then, any time the file is opened, the kernel will verify the
   file's actual digest against this signature, using the certificates
   in the ".fs-verity" keyring.

3. A new sysctl "fs.verity.require_signatures" is made available.
   When set to 1, the kernel requires that all verity files have a
   correctly signed digest as described in (2).

fs-verity file digests must be signed in the following format, which
is similar to the structure used by `FS_IOC_MEASURE_VERITY`_::

    struct fsverity_formatted_digest {
            char magic[8];                  /* must be "FSVerity" */
            __le16 digest_algorithm;
            __le16 digest_size;
            __u8 digest[];
    };

fs-verity's built-in signature verification support is meant as a
relatively simple mechanism that can be used to provide some level of
authenticity protection for verity files, as an alternative to doing
the signature verification in userspace or using IMA-appraisal.
However, with this mechanism, userspace programs still need to check
that the verity bit is set, and there is no protection against verity
files being swapped around.

Filesystem support
==================

fs-verity is supported by several filesystems, described below.  The
CONFIG_FS_VERITY kconfig option must be enabled to use fs-verity on
any of these filesystems.

``include/linux/fsverity.h`` declares the interface between the
``fs/verity/`` support layer and filesystems.  Briefly, filesystems
must provide an ``fsverity_operations`` structure that provides
methods to read and write the verity metadata to a filesystem-specific
location, including the Merkle tree blocks and
``fsverity_descriptor``.  Filesystems must also call functions in
``fs/verity/`` at certain times, such as when a file is opened or when
pages have been read into the pagecache.  (See `Verifying data`_.)

ext4
----

ext4 supports fs-verity since Linux v5.4 and e2fsprogs v1.45.2.

To create verity files on an ext4 filesystem, the filesystem must have
been formatted with ``-O verity`` or had ``tune2fs -O verity`` run on
it.  "verity" is an RO_COMPAT filesystem feature, so once set, old
kernels will only be able to mount the filesystem readonly, and old
versions of e2fsck will be unable to check the filesystem.  Moreover,
currently ext4 only supports mounting a filesystem with the "verity"
feature when its block size is equal to PAGE_SIZE (often 4096 bytes).

ext4 sets the EXT4_VERITY_FL on-disk inode flag on verity files.  It
can only be set by `FS_IOC_ENABLE_VERITY`_, and it cannot be cleared.

ext4 also supports encryption, which can be used simultaneously with
fs-verity.  In this case, the plaintext data is verified rather than
the ciphertext.  This is necessary in order to make the fs-verity file
digest meaningful, since every file is encrypted differently.

ext4 stores the verity metadata (Merkle tree and fsverity_descriptor)
past the end of the file, starting at the first 64K boundary beyond
i_size.  This approach works because (a) verity files are readonly,
and (b) pages fully beyond i_size aren't visible to userspace but can
be read/written internally by ext4 with only some relatively small
changes to ext4.  This approach avoids having to depend on the
EA_INODE feature and on rearchitecturing ext4's xattr support to
support paging multi-gigabyte xattrs into memory, and to support
encrypting xattrs.  Note that the verity metadata *must* be encrypted
when the file is, since it contains hashes of the plaintext data.

Currently, ext4 verity only supports the case where the Merkle tree
block size, filesystem block size, and page size are all the same.  It
also only supports extent-based files.

f2fs
----

f2fs supports fs-verity since Linux v5.4 and f2fs-tools v1.11.0.

To create verity files on an f2fs filesystem, the filesystem must have
been formatted with ``-O verity``.

f2fs sets the FADVISE_VERITY_BIT on-disk inode flag on verity files.
It can only be set by `FS_IOC_ENABLE_VERITY`_, and it cannot be
cleared.

Like ext4, f2fs stores the verity metadata (Merkle tree and
fsverity_descriptor) past the end of the file, starting at the first
64K boundary beyond i_size.  See explanation for ext4 above.
Moreover, f2fs supports at most 4096 bytes of xattr entries per inode
which wouldn't be enough for even a single Merkle tree block.

Currently, f2fs verity only supports a Merkle tree block size of 4096.
Also, f2fs doesn't support enabling verity on files that currently
have atomic or volatile writes pending.

btrfs
-----

btrfs supports fs-verity since Linux v5.15.  Verity-enabled inodes are
marked with a RO_COMPAT inode flag, and the verity metadata is stored
in separate btree items.

Implementation details
======================

Verifying data
--------------

fs-verity ensures that all reads of a verity file's data are verified,
regardless of which syscall is used to do the read (e.g. mmap(),
read(), pread()) and regardless of whether it's the first read or a
later read (unless the later read can return cached data that was
already verified).  Below, we describe how filesystems implement this.

Pagecache
~~~~~~~~~

For filesystems using Linux's pagecache, the ``->read_folio()`` and
``->readahead()`` methods must be modified to verify pages before they
are marked Uptodate.  Merely hooking ``->read_iter()`` would be
insufficient, since ``->read_iter()`` is not used for memory maps.

Therefore, fs/verity/ provides a function fsverity_verify_page() which
verifies a page that has been read into the pagecache of a verity
inode, but is still locked and not Uptodate, so it's not yet readable
by userspace.  As needed to do the verification,
fsverity_verify_page() will call back into the filesystem to read
Merkle tree pages via fsverity_operations::read_merkle_tree_page().

fsverity_verify_page() returns false if verification failed; in this
case, the filesystem must not set the page Uptodate.  Following this,
as per the usual Linux pagecache behavior, attempts by userspace to
read() from the part of the file containing the page will fail with
EIO, and accesses to the page within a memory map will raise SIGBUS.

fsverity_verify_page() currently only supports the case where the
Merkle tree block size is equal to PAGE_SIZE (often 4096 bytes).

In principle, fsverity_verify_page() verifies the entire path in the
Merkle tree from the data page to the root hash.  However, for
efficiency the filesystem may cache the hash pages.  Therefore,
fsverity_verify_page() only ascends the tree reading hash pages until
an already-verified hash page is seen, as indicated by the PageChecked
bit being set.  It then verifies the path to that page.

This optimization, which is also used by dm-verity, results in
excellent sequential read performance.  This is because usually (e.g.
127 in 128 times for 4K blocks and SHA-256) the hash page from the
bottom level of the tree will already be cached and checked from
reading a previous data page.  However, random reads perform worse.

Block device based filesystems
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Block device based filesystems (e.g. ext4 and f2fs) in Linux also use
the pagecache, so the above subsection applies too.  However, they
also usually read many pages from a file at once, grouped into a
structure called a "bio".  To make it easier for these types of
filesystems to support fs-verity, fs/verity/ also provides a function
fsverity_verify_bio() which verifies all pages in a bio.

ext4 and f2fs also support encryption.  If a verity file is also
encrypted, the pages must be decrypted before being verified.  To
support this, these filesystems allocate a "post-read context" for
each bio and store it in ``->bi_private``::

    struct bio_post_read_ctx {
           struct bio *bio;
           struct work_struct work;
           unsigned int cur_step;
           unsigned int enabled_steps;
    };

``enabled_steps`` is a bitmask that specifies whether decryption,
verity, or both is enabled.  After the bio completes, for each needed
postprocessing step the filesystem enqueues the bio_post_read_ctx on a
workqueue, and then the workqueue work does the decryption or
verification.  Finally, pages where no decryption or verity error
occurred are marked Uptodate, and the pages are unlocked.

On many filesystems, files can contain holes.  Normally,
``->readahead()`` simply zeroes holes and sets the corresponding pages
Uptodate; no bios are issued.  To prevent this case from bypassing
fs-verity, these filesystems use fsverity_verify_page() to verify hole
pages.

Filesystems also disable direct I/O on verity files, since otherwise
direct I/O would bypass fs-verity.

Userspace utility
=================

This document focuses on the kernel, but a userspace utility for
fs-verity can be found at:

	https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/fsverity-utils.git

See the README.md file in the fsverity-utils source tree for details,
including examples of setting up fs-verity protected files.

Tests
=====

To test fs-verity, use xfstests.  For example, using `kvm-xfstests
<https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-quickstart.md>`_::

    kvm-xfstests -c ext4,f2fs,btrfs -g verity

FAQ
===

This section answers frequently asked questions about fs-verity that
weren't already directly answered in other parts of this document.

:Q: Why isn't fs-verity part of IMA?
:A: fs-verity and IMA (Integrity Measurement Architecture) have
    different focuses.  fs-verity is a filesystem-level mechanism for
    hashing individual files using a Merkle tree.  In contrast, IMA
    specifies a system-wide policy that specifies which files are
    hashed and what to do with those hashes, such as log them,
    authenticate them, or add them to a measurement list.

    IMA supports the fs-verity hashing mechanism as an alternative
    to full file hashes, for those who want the performance and
    security benefits of the Merkle tree based hash.  However, it
    doesn't make sense to force all uses of fs-verity to be through
    IMA.  fs-verity already meets many users' needs even as a
    standalone filesystem feature, and it's testable like other
    filesystem features e.g. with xfstests.

:Q: Isn't fs-verity useless because the attacker can just modify the
    hashes in the Merkle tree, which is stored on-disk?
:A: To verify the authenticity of an fs-verity file you must verify
    the authenticity of the "fs-verity file digest", which
    incorporates the root hash of the Merkle tree.  See `Use cases`_.

:Q: Isn't fs-verity useless because the attacker can just replace a
    verity file with a non-verity one?
:A: See `Use cases`_.  In the initial use case, it's really trusted
    userspace code that authenticates the files; fs-verity is just a
    tool to do this job efficiently and securely.  The trusted
    userspace code will consider non-verity files to be inauthentic.

:Q: Why does the Merkle tree need to be stored on-disk?  Couldn't you
    store just the root hash?
:A: If the Merkle tree wasn't stored on-disk, then you'd have to
    compute the entire tree when the file is first accessed, even if
    just one byte is being read.  This is a fundamental consequence of
    how Merkle tree hashing works.  To verify a leaf node, you need to
    verify the whole path to the root hash, including the root node
    (the thing which the root hash is a hash of).  But if the root
    node isn't stored on-disk, you have to compute it by hashing its
    children, and so on until you've actually hashed the entire file.

    That defeats most of the point of doing a Merkle tree-based hash,
    since if you have to hash the whole file ahead of time anyway,
    then you could simply do sha256(file) instead.  That would be much
    simpler, and a bit faster too.

    It's true that an in-memory Merkle tree could still provide the
    advantage of verification on every read rather than just on the
    first read.  However, it would be inefficient because every time a
    hash page gets evicted (you can't pin the entire Merkle tree into
    memory, since it may be very large), in order to restore it you
    again need to hash everything below it in the tree.  This again
    defeats most of the point of doing a Merkle tree-based hash, since
    a single block read could trigger re-hashing gigabytes of data.

:Q: But couldn't you store just the leaf nodes and compute the rest?
:A: See previous answer; this really just moves up one level, since
    one could alternatively interpret the data blocks as being the
    leaf nodes of the Merkle tree.  It's true that the tree can be
    computed much faster if the leaf level is stored rather than just
    the data, but that's only because each level is less than 1% the
    size of the level below (assuming the recommended settings of
    SHA-256 and 4K blocks).  For the exact same reason, by storing
    "just the leaf nodes" you'd already be storing over 99% of the
    tree, so you might as well simply store the whole tree.

:Q: Can the Merkle tree be built ahead of time, e.g. distributed as
    part of a package that is installed to many computers?
:A: This isn't currently supported.  It was part of the original
    design, but was removed to simplify the kernel UAPI and because it
    wasn't a critical use case.  Files are usually installed once and
    used many times, and cryptographic hashing is somewhat fast on
    most modern processors.

:Q: Why doesn't fs-verity support writes?
:A: Write support would be very difficult and would require a
    completely different design, so it's well outside the scope of
    fs-verity.  Write support would require:

    - A way to maintain consistency between the data and hashes,
      including all levels of hashes, since corruption after a crash
      (especially of potentially the entire file!) is unacceptable.
      The main options for solving this are data journalling,
      copy-on-write, and log-structured volume.  But it's very hard to
      retrofit existing filesystems with new consistency mechanisms.
      Data journalling is available on ext4, but is very slow.

    - Rebuilding the Merkle tree after every write, which would be
      extremely inefficient.  Alternatively, a different authenticated
      dictionary structure such as an "authenticated skiplist" could
      be used.  However, this would be far more complex.

    Compare it to dm-verity vs. dm-integrity.  dm-verity is very
    simple: the kernel just verifies read-only data against a
    read-only Merkle tree.  In contrast, dm-integrity supports writes
    but is slow, is much more complex, and doesn't actually support
    full-device authentication since it authenticates each sector
    independently, i.e. there is no "root hash".  It doesn't really
    make sense for the same device-mapper target to support these two
    very different cases; the same applies to fs-verity.

:Q: Since verity files are immutable, why isn't the immutable bit set?
:A: The existing "immutable" bit (FS_IMMUTABLE_FL) already has a
    specific set of semantics which not only make the file contents
    read-only, but also prevent the file from being deleted, renamed,
    linked to, or having its owner or mode changed.  These extra
    properties are unwanted for fs-verity, so reusing the immutable
    bit isn't appropriate.

:Q: Why does the API use ioctls instead of setxattr() and getxattr()?
:A: Abusing the xattr interface for basically arbitrary syscalls is
    heavily frowned upon by most of the Linux filesystem developers.
    An xattr should really just be an xattr on-disk, not an API to
    e.g. magically trigger construction of a Merkle tree.

:Q: Does fs-verity support remote filesystems?
:A: So far all filesystems that have implemented fs-verity support are
    local filesystems, but in principle any filesystem that can store
    per-file verity metadata can support fs-verity, regardless of
    whether it's local or remote.  Some filesystems may have fewer
    options of where to store the verity metadata; one possibility is
    to store it past the end of the file and "hide" it from userspace
    by manipulating i_size.  The data verification functions provided
    by ``fs/verity/`` also assume that the filesystem uses the Linux
    pagecache, but both local and remote filesystems normally do so.

:Q: Why is anything filesystem-specific at all?  Shouldn't fs-verity
    be implemented entirely at the VFS level?
:A: There are many reasons why this is not possible or would be very
    difficult, including the following:

    - To prevent bypassing verification, pages must not be marked
      Uptodate until they've been verified.  Currently, each
      filesystem is responsible for marking pages Uptodate via
      ``->readahead()``.  Therefore, currently it's not possible for
      the VFS to do the verification on its own.  Changing this would
      require significant changes to the VFS and all filesystems.

    - It would require defining a filesystem-independent way to store
      the verity metadata.  Extended attributes don't work for this
      because (a) the Merkle tree may be gigabytes, but many
      filesystems assume that all xattrs fit into a single 4K
      filesystem block, and (b) ext4 and f2fs encryption doesn't
      encrypt xattrs, yet the Merkle tree *must* be encrypted when the
      file contents are, because it stores hashes of the plaintext
      file contents.

      So the verity metadata would have to be stored in an actual
      file.  Using a separate file would be very ugly, since the
      metadata is fundamentally part of the file to be protected, and
      it could cause problems where users could delete the real file
      but not the metadata file or vice versa.  On the other hand,
      having it be in the same file would break applications unless
      filesystems' notion of i_size were divorced from the VFS's,
      which would be complex and require changes to all filesystems.

    - It's desirable that FS_IOC_ENABLE_VERITY uses the filesystem's
      transaction mechanism so that either the file ends up with
      verity enabled, or no changes were made.  Allowing intermediate
      states to occur after a crash may cause problems.

repo_name: angery/linux


code:
.. SPDX-License-Identifier: GPL-2.0

==============
Fuse I/O Modes
==============

Fuse supports the following I/O modes:

- direct-io
- cached
  + write-through
  + writeback-cache

The direct-io mode can be selected with the FOPEN_DIRECT_IO flag in the
FUSE_OPEN reply.

In direct-io mode the page cache is completely bypassed for reads and writes.
No read-ahead takes place. Shared mmap is disabled.

In cached mode reads may be satisfied from the page cache, and data may be
read-ahead by the kernel to fill the cache.  The cache is always kept consistent
after any writes to the file.  All mmap modes are supported.

The cached mode has two sub modes controlling how writes are handled.  The
write-through mode is the default and is supported on all kernels.  The
writeback-cache mode may be selected by the FUSE_WRITEBACK_CACHE flag in the
FUSE_INIT reply.

In write-through mode each write is immediately sent to userspace as one or more
WRITE requests, as well as updating any cached pages (and caching previously
uncached, but fully written pages).  No READ requests are ever sent for writes,
so when an uncached page is partially written, the page is discarded.

In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flag) writes go to
the cache only, which means that the write(2) syscall can often complete very
fast.  Dirty pages are written back implicitly (background writeback or page
reclaim on memory pressure) or explicitly (invoked by close(2), fsync(2) and
when the last ref to the file is being released on munmap(2)).  This mode
assumes that all changes to the filesystem go through the FUSE kernel module
(size and atime/ctime/mtime attributes are kept up-to-date by the kernel), so
it's generally not suitable for network filesystems.  If a partial page is
written, then the page needs to be first read from userspace.  This means, that
even for files opened for O_WRONLY it is possible that READ requests will be
generated by the kernel.


code:
.. SPDX-License-Identifier: GPL-2.0

====
FUSE
====

Definitions
===========

Userspace filesystem:
  A filesystem in which data and metadata are provided by an ordinary
  userspace process.  The filesystem can be accessed normally through
  the kernel interface.

Filesystem daemon:
  The process(es) providing the data and metadata of the filesystem.

Non-privileged mount (or user mount):
  A userspace filesystem mounted by a non-privileged (non-root) user.
  The filesystem daemon is running with the privileges of the mounting
  user.  NOTE: this is not the same as mounts allowed with the "user"
  option in /etc/fstab, which is not discussed here.

Filesystem connection:
  A connection between the filesystem daemon and the kernel.  The
  connection exists until either the daemon dies, or the filesystem is
  umounted.  Note that detaching (or lazy umounting) the filesystem
  does *not* break the connection, in this case it will exist until
  the last reference to the filesystem is released.

Mount owner:
  The user who does the mounting.

User:
  The user who is performing filesystem operations.

What is FUSE?
=============

FUSE is a userspace filesystem framework.  It consists of a kernel
module (fuse.ko), a userspace library (libfuse.*) and a mount utility
(fusermount).

One of the most important features of FUSE is allowing secure,
non-privileged mounts.  This opens up new possibilities for the use of
filesystems.  A good example is sshfs: a secure network filesystem
using the sftp protocol.

The userspace library and utilities are available from the
`FUSE homepage: <https://github.com/libfuse/>`_

Filesystem type
===============

The filesystem type given to mount(2) can be one of the following:

    fuse
      This is the usual way to mount a FUSE filesystem.  The first
      argument of the mount system call may contain an arbitrary string,
      which is not interpreted by the kernel.

    fuseblk
      The filesystem is block device based.  The first argument of the
      mount system call is interpreted as the name of the device.

Mount options
=============

fd=N
  The file descriptor to use for communication between the userspace
  filesystem and the kernel.  The file descriptor must have been
  obtained by opening the FUSE device ('/dev/fuse').

rootmode=M
  The file mode of the filesystem's root in octal representation.

user_id=N
  The numeric user id of the mount owner.

group_id=N
  The numeric group id of the mount owner.

default_permissions
  By default FUSE doesn't check file access permissions, the
  filesystem is free to implement its access policy or leave it to
  the underlying file access mechanism (e.g. in case of network
  filesystems).  This option enables permission checking, restricting
  access based on file mode.  It is usually useful together with the
  'allow_other' mount option.

allow_other
  This option overrides the security measure restricting file access
  to the user mounting the filesystem.  This option is by default only
  allowed to root, but this restriction can be removed with a
  (userspace) configuration option.

max_read=N
  With this option the maximum size of read operations can be set.
  The default is infinite.  Note that the size of read requests is
  limited anyway to 32 pages (which is 128kbyte on i386).

blksize=N
  Set the block size for the filesystem.  The default is 512.  This
  option is only valid for 'fuseblk' type mounts.

Control filesystem
==================

There's a control filesystem for FUSE, which can be mounted by::

  mount -t fusectl none /sys/fs/fuse/connections

Mounting it under the '/sys/fs/fuse/connections' directory makes it
backwards compatible with earlier versions.

Under the fuse control filesystem each connection has a directory
named by a unique number.

For each connection the following files exist within this directory:

	waiting
	  The number of requests which are waiting to be transferred to
	  userspace or being processed by the filesystem daemon.  If there is
	  no filesystem activity and 'waiting' is non-zero, then the
	  filesystem is hung or deadlocked.

	abort
	  Writing anything into this file will abort the filesystem
	  connection.  This means that all waiting requests will be aborted an
	  error returned for all aborted and new requests.

Only the owner of the mount may read or write these files.

Interrupting filesystem operations
##################################

If a process issuing a FUSE filesystem request is interrupted, the
following will happen:

  -  If the request is not yet sent to userspace AND the signal is
     fatal (SIGKILL or unhandled fatal signal), then the request is
     dequeued and returns immediately.

  -  If the request is not yet sent to userspace AND the signal is not
     fatal, then an interrupted flag is set for the request.  When
     the request has been successfully transferred to userspace and
     this flag is set, an INTERRUPT request is queued.

  -  If the request is already sent to userspace, then an INTERRUPT
     request is queued.

INTERRUPT requests take precedence over other requests, so the
userspace filesystem will receive queued INTERRUPTs before any others.

The userspace filesystem may ignore the INTERRUPT requests entirely,
or may honor them by sending a reply to the *original* request, with
the error set to EINTR.

It is also possible that there's a race between processing the
original request and its INTERRUPT request.  There are two possibilities:

  1. The INTERRUPT request is processed before the original request is
     processed

  2. The INTERRUPT request is processed after the original request has
     been answered

If the filesystem cannot find the original request, it should wait for
some timeout and/or a number of new requests to arrive, after which it
should reply to the INTERRUPT request with an EAGAIN error.  In case
1) the INTERRUPT request will be requeued.  In case 2) the INTERRUPT
reply will be ignored.

Aborting a filesystem connection
================================

It is possible to get into certain situations where the filesystem is
not responding.  Reasons for this may be:

  a) Broken userspace filesystem implementation

  b) Network connection down

  c) Accidental deadlock

  d) Malicious deadlock

(For more on c) and d) see later sections)

In either of these cases it may be useful to abort the connection to
the filesystem.  There are several ways to do this:

  - Kill the filesystem daemon.  Works in case of a) and b)

  - Kill the filesystem daemon and all users of the filesystem.  Works
    in all cases except some malicious deadlocks

  - Use forced umount (umount -f).  Works in all cases but only if
    filesystem is still attached (it hasn't been lazy unmounted)

  - Abort filesystem through the FUSE control filesystem.  Most
    powerful method, always works.

How do non-privileged mounts work?
==================================

Since the mount() system call is a privileged operation, a helper
program (fusermount) is needed, which is installed setuid root.

The implication of providing non-privileged mounts is that the mount
owner must not be able to use this capability to compromise the
system.  Obvious requirements arising from this are:

 A) mount owner should not be able to get elevated privileges with the
    help of the mounted filesystem

 B) mount owner should not get illegitimate access to information from
    other users' and the super user's processes

 C) mount owner should not be able to induce undesired behavior in
    other users' or the super user's processes

How are requirements fulfilled?
===============================

 A) The mount owner could gain elevated privileges by either:

    1. creating a filesystem containing a device file, then opening this device

    2. creating a filesystem containing a suid or sgid application, then executing this application

    The solution is not to allow opening device files and ignore
    setuid and setgid bits when executing programs.  To ensure this
    fusermount always adds "nosuid" and "nodev" to the mount options
    for non-privileged mounts.

 B) If another user is accessing files or directories in the
    filesystem, the filesystem daemon serving requests can record the
    exact sequence and timing of operations performed.  This
    information is otherwise inaccessible to the mount owner, so this
    counts as an information leak.

    The solution to this problem will be presented in point 2) of C).

 C) There are several ways in which the mount owner can induce
    undesired behavior in other users' processes, such as:

     1) mounting a filesystem over a file or directory which the mount
        owner could otherwise not be able to modify (or could only
        make limited modifications).

        This is solved in fusermount, by checking the access
        permissions on the mountpoint and only allowing the mount if
        the mount owner can do unlimited modification (has write
        access to the mountpoint, and mountpoint is not a "sticky"
        directory)

     2) Even if 1) is solved the mount owner can change the behavior
        of other users' processes.

         i) It can slow down or indefinitely delay the execution of a
            filesystem operation creating a DoS against the user or the
            whole system.  For example a suid application locking a
            system file, and then accessing a file on the mount owner's
            filesystem could be stopped, and thus causing the system
            file to be locked forever.

         ii) It can present files or directories of unlimited length, or
             directory structures of unlimited depth, possibly causing a
             system process to eat up diskspace, memory or other
             resources, again causing *DoS*.

	The solution to this as well as B) is not to allow processes
	to access the filesystem, which could otherwise not be
	monitored or manipulated by the mount owner.  Since if the
	mount owner can ptrace a process, it can do all of the above
	without using a FUSE mount, the same criteria as used in
	ptrace can be used to check if a process is allowed to access
	the filesystem or not.

	Note that the *ptrace* check is not strictly necessary to
	prevent C/2/i, it is enough to check if mount owner has enough
	privilege to send signal to the process accessing the
	filesystem, since *SIGSTOP* can be used to get a similar effect.

I think these limitations are unacceptable?
===========================================

If a sysadmin trusts the users enough, or can ensure through other
measures, that system processes will never enter non-privileged
mounts, it can relax the last limitation in several ways:

  - With the 'user_allow_other' config option. If this config option is
    set, the mounting user can add the 'allow_other' mount option which
    disables the check for other users' processes.

    User namespaces have an unintuitive interaction with 'allow_other':
    an unprivileged user - normally restricted from mounting with
    'allow_other' - could do so in a user namespace where they're
    privileged. If any process could access such an 'allow_other' mount
    this would give the mounting user the ability to manipulate
    processes in user namespaces where they're unprivileged. For this
    reason 'allow_other' restricts access to users in the same userns
    or a descendant.

  - With the 'allow_sys_admin_access' module option. If this option is
    set, super user's processes have unrestricted access to mounts
    irrespective of allow_other setting or user namespace of the
    mounting user.

Note that both of these relaxations expose the system to potential
information leak or *DoS* as described in points B and C/2/i-ii in the
preceding section.

Kernel - userspace interface
============================

The following diagram shows how a filesystem operation (in this
example unlink) is performed in FUSE. ::

 |  "rm /mnt/fuse/file"               |  FUSE filesystem daemon
 |                                    |
 |                                    |  >sys_read()
 |                                    |    >fuse_dev_read()
 |                                    |      >request_wait()
 |                                    |        [sleep on fc->waitq]
 |                                    |
 |  >sys_unlink()                     |
 |    >fuse_unlink()                  |
 |      [get request from             |
 |       fc->unused_list]             |
 |      >request_send()               |
 |        [queue req on fc->pending]  |
 |        [wake up fc->waitq]         |        [woken up]
 |        >request_wait_answer()      |
 |          [sleep on req->waitq]     |
 |                                    |      <request_wait()
 |                                    |      [remove req from fc->pending]
 |                                    |      [copy req to read buffer]
 |                                    |      [add req to fc->processing]
 |                                    |    <fuse_dev_read()
 |                                    |  <sys_read()
 |                                    |
 |                                    |  [perform unlink]
 |                                    |
 |                                    |  >sys_write()
 |                                    |    >fuse_dev_write()
 |                                    |      [look up req in fc->processing]
 |                                    |      [remove from fc->processing]
 |                                    |      [copy write buffer to req]
 |          [woken up]                |      [wake up req->waitq]
 |                                    |    <fuse_dev_write()
 |                                    |  <sys_write()
 |        <request_wait_answer()      |
 |      <request_send()               |
 |      [add request to               |
 |       fc->unused_list]             |
 |    <fuse_unlink()                  |
 |  <sys_unlink()                     |

.. note:: Everything in the description above is greatly simplified

There are a couple of ways in which to deadlock a FUSE filesystem.
Since we are talking about unprivileged userspace programs,
something must be done about these.

**Scenario 1 -  Simple deadlock**::

 |  "rm /mnt/fuse/file"               |  FUSE filesystem daemon
 |                                    |
 |  >sys_unlink("/mnt/fuse/file")     |
 |    [acquire inode semaphore        |
 |     for "file"]                    |
 |    >fuse_unlink()                  |
 |      [sleep on req->waitq]         |
 |                                    |  <sys_read()
 |                                    |  >sys_unlink("/mnt/fuse/file")
 |                                    |    [acquire inode semaphore
 |                                    |     for "file"]
 |                                    |    *DEADLOCK*

The solution for this is to allow the filesystem to be aborted.

**Scenario 2 - Tricky deadlock**

This one needs a carefully crafted filesystem.  It's a variation on
the above, only the call back to the filesystem is not explicit,
but is caused by a pagefault. ::

 |  Kamikaze filesystem thread 1      |  Kamikaze filesystem thread 2
 |                                    |
 |  [fd = open("/mnt/fuse/file")]     |  [request served normally]
 |  [mmap fd to 'addr']               |
 |  [close fd]                        |  [FLUSH triggers 'magic' flag]
 |  [read a byte from addr]           |
 |    >do_page_fault()                |
 |      [find or create page]         |
 |      [lock page]                   |
 |      >fuse_readpage()              |
 |         [queue READ request]       |
 |         [sleep on req->waitq]      |
 |                                    |  [read request to buffer]
 |                                    |  [create reply header before addr]
 |                                    |  >sys_write(addr - headerlength)
 |                                    |    >fuse_dev_write()
 |                                    |      [look up req in fc->processing]
 |                                    |      [remove from fc->processing]
 |                                    |      [copy write buffer to req]
 |                                    |        >do_page_fault()
 |                                    |           [find or create page]
 |                                    |           [lock page]
 |                                    |           * DEADLOCK *

The solution is basically the same as above.

An additional problem is that while the write buffer is being copied
to the request, the request must not be interrupted/aborted.  This is
because the destination address of the copy may not be valid after the
request has returned.

This is solved with doing the copy atomically, and allowing abort
while the page(s) belonging to the write buffer are faulted with
get_user_pages().  The 'req->locked' flag indicates when the copy is
taking place, and abort is delayed until this flag is unset.


code:
.. SPDX-License-Identifier: GPL-2.0

============================
Glock internal locking rules
============================

This documents the basic principles of the glock state machine
internals. Each glock (struct gfs2_glock in fs/gfs2/incore.h)
has two main (internal) locks:

 1. A spinlock (gl_lockref.lock) which protects the internal state such
    as gl_state, gl_target and the list of holders (gl_holders)
 2. A non-blocking bit lock, GLF_LOCK, which is used to prevent other
    threads from making calls to the DLM, etc. at the same time. If a
    thread takes this lock, it must then call run_queue (usually via the
    workqueue) when it releases it in order to ensure any pending tasks
    are completed.

The gl_holders list contains all the queued lock requests (not
just the holders) associated with the glock. If there are any
held locks, then they will be contiguous entries at the head
of the list. Locks are granted in strictly the order that they
are queued, except for those marked LM_FLAG_PRIORITY which are
used only during recovery, and even then only for journal locks.

There are three lock states that users of the glock layer can request,
namely shared (SH), deferred (DF) and exclusive (EX). Those translate
to the following DLM lock modes:

==========	====== =====================================================
Glock mode      DLM    lock mode
==========	====== =====================================================
    UN          IV/NL  Unlocked (no DLM lock associated with glock) or NL
    SH          PR     (Protected read)
    DF          CW     (Concurrent write)
    EX          EX     (Exclusive)
==========	====== =====================================================

Thus DF is basically a shared mode which is incompatible with the "normal"
shared lock mode, SH. In GFS2 the DF mode is used exclusively for direct I/O
operations. The glocks are basically a lock plus some routines which deal
with cache management. The following rules apply for the cache:

==========      ==========   ==============   ==========   ==============
Glock mode      Cache data   Cache Metadata   Dirty Data   Dirty Metadata
==========      ==========   ==============   ==========   ==============
    UN             No              No             No            No
    SH             Yes             Yes            No            No
    DF             No              Yes            No            No
    EX             Yes             Yes            Yes           Yes
==========      ==========   ==============   ==========   ==============

These rules are implemented using the various glock operations which
are defined for each type of glock. Not all types of glocks use
all the modes. Only inode glocks use the DF mode for example.

Table of glock operations and per type constants:

=============      =============================================================
Field              Purpose
=============      =============================================================
go_xmote_th        Called before remote state change (e.g. to sync dirty data)
go_xmote_bh        Called after remote state change (e.g. to refill cache)
go_inval           Called if remote state change requires invalidating the cache
go_demote_ok       Returns boolean value of whether its ok to demote a glock
                   (e.g. checks timeout, and that there is no cached data)
go_lock            Called for the first local holder of a lock
go_unlock          Called on the final local unlock of a lock
go_dump            Called to print content of object for debugfs file, or on
                   error to dump glock to the log.
go_type            The type of the glock, ``LM_TYPE_*``
go_callback	   Called if the DLM sends a callback to drop this lock
go_flags	   GLOF_ASPACE is set, if the glock has an address space
                   associated with it
=============      =============================================================

The minimum hold time for each lock is the time after a remote lock
grant for which we ignore remote demote requests. This is in order to
prevent a situation where locks are being bounced around the cluster
from node to node with none of the nodes making any progress. This
tends to show up most with shared mmaped files which are being written
to by multiple nodes. By delaying the demotion in response to a
remote callback, that gives the userspace program time to make
some progress before the pages are unmapped.

There is a plan to try and remove the go_lock and go_unlock callbacks
if possible, in order to try and speed up the fast path though the locking.
Also, eventually we hope to make the glock "EX" mode locally shared
such that any local locking will be done with the i_mutex as required
rather than via the glock.

Locking rules for glock operations:

=============    ======================    =============================
Operation        GLF_LOCK bit lock held    gl_lockref.lock spinlock held
=============    ======================    =============================
go_xmote_th           Yes                       No
go_xmote_bh           Yes                       No
go_inval              Yes                       No
go_demote_ok          Sometimes                 Yes
go_lock               Yes                       No
go_unlock             Yes                       No
go_dump               Sometimes                 Yes
go_callback           Sometimes (N/A)           Yes
=============    ======================    =============================

.. Note::

   Operations must not drop either the bit lock or the spinlock
   if its held on entry. go_dump and do_demote_ok must never block.
   Note that go_dump will only be called if the glock's state
   indicates that it is caching uptodate data.

Glock locking order within GFS2:

 1. i_rwsem (if required)
 2. Rename glock (for rename only)
 3. Inode glock(s)
    (Parents before children, inodes at "same level" with same parent in
    lock number order)
 4. Rgrp glock(s) (for (de)allocation operations)
 5. Transaction glock (via gfs2_trans_begin) for non-read operations
 6. i_rw_mutex (if required)
 7. Page lock  (always last, very important!)

There are two glocks per inode. One deals with access to the inode
itself (locking order as above), and the other, known as the iopen
glock is used in conjunction with the i_nlink field in the inode to
determine the lifetime of the inode in question. Locking of inodes
is on a per-inode basis. Locking of rgrps is on a per rgrp basis.
In general we prefer to lock local locks prior to cluster locks.

Glock Statistics
----------------

The stats are divided into two sets: those relating to the
super block and those relating to an individual glock. The
super block stats are done on a per cpu basis in order to
try and reduce the overhead of gathering them. They are also
further divided by glock type. All timings are in nanoseconds.

In the case of both the super block and glock statistics,
the same information is gathered in each case. The super
block timing statistics are used to provide default values for
the glock timing statistics, so that newly created glocks
should have, as far as possible, a sensible starting point.
The per-glock counters are initialised to zero when the
glock is created. The per-glock statistics are lost when
the glock is ejected from memory.

The statistics are divided into three pairs of mean and
variance, plus two counters. The mean/variance pairs are
smoothed exponential estimates and the algorithm used is
one which will be very familiar to those used to calculation
of round trip times in network code. See "TCP/IP Illustrated,
Volume 1", W. Richard Stevens, sect 21.3, "Round-Trip Time Measurement",
p. 299 and onwards. Also, Volume 2, Sect. 25.10, p. 838 and onwards.
Unlike the TCP/IP Illustrated case, the mean and variance are
not scaled, but are in units of integer nanoseconds.

The three pairs of mean/variance measure the following
things:

 1. DLM lock time (non-blocking requests)
 2. DLM lock time (blocking requests)
 3. Inter-request time (again to the DLM)

A non-blocking request is one which will complete right
away, whatever the state of the DLM lock in question. That
currently means any requests when (a) the current state of
the lock is exclusive, i.e. a lock demotion (b) the requested
state is either null or unlocked (again, a demotion) or (c) the
"try lock" flag is set. A blocking request covers all the other
lock requests.

There are two counters. The first is there primarily to show
how many lock requests have been made, and thus how much data
has gone into the mean/variance calculations. The other counter
is counting queuing of holders at the top layer of the glock
code. Hopefully that number will be a lot larger than the number
of dlm lock requests issued.

So why gather these statistics? There are several reasons
we'd like to get a better idea of these timings:

1. To be able to better set the glock "min hold time"
2. To spot performance issues more easily
3. To improve the algorithm for selecting resource groups for
   allocation (to base it on lock wait time, rather than blindly
   using a "try lock")

Due to the smoothing action of the updates, a step change in
some input quantity being sampled will only fully be taken
into account after 8 samples (or 4 for the variance) and this
needs to be carefully considered when interpreting the
results.

Knowing both the time it takes a lock request to complete and
the average time between lock requests for a glock means we
can compute the total percentage of the time for which the
node is able to use a glock vs. time that the rest of the
cluster has its share. That will be very useful when setting
the lock min hold time.

Great care has been taken to ensure that we
measure exactly the quantities that we want, as accurately
as possible. There are always inaccuracies in any
measuring system, but I hope this is as accurate as we
can reasonably make it.

Per sb stats can be found here::

    /sys/kernel/debug/gfs2/<fsname>/sbstats

Per glock stats can be found here::

    /sys/kernel/debug/gfs2/<fsname>/glstats

Assuming that debugfs is mounted on /sys/kernel/debug and also
that <fsname> is replaced with the name of the gfs2 filesystem
in question.

The abbreviations used in the output as are follows:

=========  ================================================================
srtt       Smoothed round trip time for non blocking dlm requests
srttvar    Variance estimate for srtt
srttb      Smoothed round trip time for (potentially) blocking dlm requests
srttvarb   Variance estimate for srttb
sirt       Smoothed inter request time (for dlm requests)
sirtvar    Variance estimate for sirt
dlm        Number of dlm requests made (dcnt in glstats file)
queue      Number of glock requests queued (qcnt in glstats file)
=========  ================================================================

The sbstats file contains a set of these stats for each glock type (so 8 lines
for each type) and for each cpu (one column per cpu). The glstats file contains
a set of these stats for each glock in a similar format to the glocks file, but
using the format mean/variance for each of the timing stats.

The gfs2_glock_lock_time tracepoint prints out the current values of the stats
for the glock in question, along with some addition information on each dlm
reply that is received:

======   =======================================
status   The status of the dlm request
flags    The dlm request flags
tdiff    The time taken by this specific request
======   =======================================

(remaining fields as per above list)


code:
.. SPDX-License-Identifier: GPL-2.0

================
uevents and GFS2
================

During the lifetime of a GFS2 mount, a number of uevents are generated.
This document explains what the events are and what they are used
for (by gfs_controld in gfs2-utils).

A list of GFS2 uevents
======================

1. ADD
------

The ADD event occurs at mount time. It will always be the first
uevent generated by the newly created filesystem. If the mount
is successful, an ONLINE uevent will follow.  If it is not successful
then a REMOVE uevent will follow.

The ADD uevent has two environment variables: SPECTATOR=[0|1]
and RDONLY=[0|1] that specify the spectator status (a read-only mount
with no journal assigned), and read-only (with journal assigned) status
of the filesystem respectively.

2. ONLINE
---------

The ONLINE uevent is generated after a successful mount or remount. It
has the same environment variables as the ADD uevent. The ONLINE
uevent, along with the two environment variables for spectator and
RDONLY are a relatively recent addition (2.6.32-rc+) and will not
be generated by older kernels.

3. CHANGE
---------

The CHANGE uevent is used in two places. One is when reporting the
successful mount of the filesystem by the first node (FIRSTMOUNT=Done).
This is used as a signal by gfs_controld that it is then ok for other
nodes in the cluster to mount the filesystem.

The other CHANGE uevent is used to inform of the completion
of journal recovery for one of the filesystems journals. It has
two environment variables, JID= which specifies the journal id which
has just been recovered, and RECOVERY=[Done|Failed] to indicate the
success (or otherwise) of the operation. These uevents are generated
for every journal recovered, whether it is during the initial mount
process or as the result of gfs_controld requesting a specific journal
recovery via the /sys/fs/gfs2/<fsname>/lock_module/recovery file.

Because the CHANGE uevent was used (in early versions of gfs_controld)
without checking the environment variables to discover the state, we
cannot add any more functions to it without running the risk of
someone using an older version of the user tools and breaking their
cluster. For this reason the ONLINE uevent was used when adding a new
uevent for a successful mount or remount.

4. OFFLINE
----------

The OFFLINE uevent is only generated due to filesystem errors and is used
as part of the "withdraw" mechanism. Currently this doesn't give any
information about what the error is, which is something that needs to
be fixed.

5. REMOVE
---------

The REMOVE uevent is generated at the end of an unsuccessful mount
or at the end of a umount of the filesystem. All REMOVE uevents will
have been preceded by at least an ADD uevent for the same filesystem,
and unlike the other uevents is generated automatically by the kernel's
kobject subsystem.

Information common to all GFS2 uevents (uevent environment variables)
=====================================================================

1. LOCKTABLE=
--------------

The LOCKTABLE is a string, as supplied on the mount command
line (locktable=) or via fstab. It is used as a filesystem label
as well as providing the information for a lock_dlm mount to be
able to join the cluster.

2. LOCKPROTO=
-------------

The LOCKPROTO is a string, and its value depends on what is set
on the mount command line, or via fstab. It will be either
lock_nolock or lock_dlm. In the future other lock managers
may be supported.

3. JOURNALID=
-------------

If a journal is in use by the filesystem (journals are not
assigned for spectator mounts) then this will give the
numeric journal id in all GFS2 uevents.

4. UUID=
--------

With recent versions of gfs2-utils, mkfs.gfs2 writes a UUID
into the filesystem superblock. If it exists, this will
be included in every uevent relating to the filesystem.


code:
.. SPDX-License-Identifier: GPL-2.0

====================
Global File System 2
====================

GFS2 is a cluster file system. It allows a cluster of computers to
simultaneously use a block device that is shared between them (with FC,
iSCSI, NBD, etc).  GFS2 reads and writes to the block device like a local
file system, but also uses a lock module to allow the computers coordinate
their I/O so file system consistency is maintained.  One of the nifty
features of GFS2 is perfect consistency -- changes made to the file system
on one machine show up immediately on all other machines in the cluster.

GFS2 uses interchangeable inter-node locking mechanisms, the currently
supported mechanisms are:

  lock_nolock
    - allows GFS2 to be used as a local file system

  lock_dlm
    - uses the distributed lock manager (dlm) for inter-node locking.
      The dlm is found at linux/fs/dlm/

lock_dlm depends on user space cluster management systems found
at the URL above.

To use GFS2 as a local file system, no external clustering systems are
needed, simply::

  $ mkfs -t gfs2 -p lock_nolock -j 1 /dev/block_device
  $ mount -t gfs2 /dev/block_device /dir

The gfs2-utils package is required on all cluster nodes and, for lock_dlm, you
will also need the dlm and corosync user space utilities configured as per the
documentation.

gfs2-utils can be found at https://pagure.io/gfs2-utils

GFS2 is not on-disk compatible with previous versions of GFS, but it
is pretty close.

The following man pages are available from gfs2-utils:

  ============		=============================================
  fsck.gfs2		to repair a filesystem
  gfs2_grow		to expand a filesystem online
  gfs2_jadd		to add journals to a filesystem online
  tunegfs2		to manipulate, examine and tune a filesystem
  gfs2_convert		to convert a gfs filesystem to GFS2 in-place
  mkfs.gfs2		to make a filesystem
  ============		=============================================


code:
.. SPDX-License-Identifier: GPL-2.0

==================================
Macintosh HFS Filesystem for Linux
==================================

.. Note:: This filesystem doesn't have a maintainer.

HFS stands for ``Hierarchical File System`` and is the filesystem used
by the Mac Plus and all later Macintosh models.  Earlier Macintosh
models used MFS (``Macintosh File System``), which is not supported,
MacOS 8.1 and newer support a filesystem called HFS+ that's similar to
HFS but is extended in various areas.  Use the hfsplus filesystem driver
to access such filesystems from Linux.

Mount options
=============

When mounting an HFS filesystem, the following options are accepted:

  creator=cccc, type=cccc
	Specifies the creator/type values as shown by the MacOS finder
	used for creating new files.  Default values: '????'.

  uid=n, gid=n
  	Specifies the user/group that owns all files on the filesystems.
	Default:  user/group id of the mounting process.

  dir_umask=n, file_umask=n, umask=n
	Specifies the umask used for all files , all directories or all
	files and directories.  Defaults to the umask of the mounting process.

  session=n
  	Select the CDROM session to mount as HFS filesystem.  Defaults to
	leaving that decision to the CDROM driver.  This option will fail
	with anything but a CDROM as underlying devices.

  part=n
  	Select partition number n from the devices.  Does only makes
	sense for CDROMS because they can't be partitioned under Linux.
	For disk devices the generic partition parsing code does this
	for us.  Defaults to not parsing the partition table at all.

  quiet
  	Ignore invalid mount options instead of complaining.

Writing to HFS Filesystems
==========================

HFS is not a UNIX filesystem, thus it does not have the usual features you'd
expect:

 * You can't modify the set-uid, set-gid, sticky or executable bits or the uid
   and gid of files.
 * You can't create hard- or symlinks, device files, sockets or FIFOs.

HFS does on the other have the concepts of multiple forks per file.  These
non-standard forks are represented as hidden additional files in the normal
filesystems namespace which is kind of a cludge and makes the semantics for
the a little strange:

 * You can't create, delete or rename resource forks of files or the
   Finder's metadata.
 * They are however created (with default values), deleted and renamed
   along with the corresponding data fork or directory.
 * Copying files to a different filesystem will loose those attributes
   that are essential for MacOS to work.

Creating HFS filesystems
========================

The hfsutils package from Robert Leslie contains a program called
hformat that can be used to create HFS filesystem. See
<https://www.mars.org/home/rob/proj/hfs/> for details.

Credits
=======

The HFS drivers was written by Paul H. Hargrovea (hargrove@sccm.Stanford.EDU).
Roman Zippel (roman@ardistech.com) rewrote large parts of the code and brought
in btree routines derived from Brad Boyer's hfsplus driver.


code:
.. SPDX-License-Identifier: GPL-2.0

======================================
Macintosh HFSPlus Filesystem for Linux
======================================

HFSPlus is a filesystem first introduced in MacOS 8.1.
HFSPlus has several extensions to HFS, including 32-bit allocation
blocks, 255-character unicode filenames, and file sizes of 2^63 bytes.

Mount options
=============

When mounting an HFSPlus filesystem, the following options are accepted:

  creator=cccc, type=cccc
	Specifies the creator/type values as shown by the MacOS finder
	used for creating new files.  Default values: '????'.

  uid=n, gid=n
	Specifies the user/group that owns all files on the filesystem
	that have uninitialized permissions structures.
	Default:  user/group id of the mounting process.

  umask=n
	Specifies the umask (in octal) used for files and directories
	that have uninitialized permissions structures.
	Default:  umask of the mounting process.

  session=n
	Select the CDROM session to mount as HFSPlus filesystem.  Defaults to
	leaving that decision to the CDROM driver.  This option will fail
	with anything but a CDROM as underlying devices.

  part=n
	Select partition number n from the devices.  This option only makes
	sense for CDROMs because they can't be partitioned under Linux.
	For disk devices the generic partition parsing code does this
	for us.  Defaults to not parsing the partition table at all.

  decompose
	Decompose file name characters.

  nodecompose
	Do not decompose file name characters.

  force
	Used to force write access to volumes that are marked as journalled
	or locked.  Use at your own risk.

  nls=cccc
	Encoding to use when presenting file names.

References
==========

kernel source:		<file:fs/hfsplus>

Apple Technote 1150	https://developer.apple.com/legacy/library/technotes/tn/tn1150.html


code:
.. SPDX-License-Identifier: GPL-2.0

====================
Read/Write HPFS 2.09
====================

1998-2004, Mikulas Patocka

:email: mikulas@artax.karlin.mff.cuni.cz
:homepage: https://artax.karlin.mff.cuni.cz/~mikulas/vyplody/hpfs/index-e.cgi

Credits
=======
Chris Smith, 1993, original read-only HPFS, some code and hpfs structures file
	is taken from it

Jacques Gelinas, MSDos mmap, Inspired by fs/nfs/mmap.c (Jon Tombs 15 Aug 1993)

Werner Almesberger, 1992, 1993, MSDos option parser & CR/LF conversion

Mount options

uid=xxx,gid=xxx,umask=xxx (default uid=gid=0 umask=default_system_umask)
	Set owner/group/mode for files that do not have it specified in extended
	attributes. Mode is inverted umask - for example umask 027 gives owner
	all permission, group read permission and anybody else no access. Note
	that for files mode is anded with 0666. If you want files to have 'x'
	rights, you must use extended attributes.
case=lower,asis (default asis)
	File name lowercasing in readdir.
conv=binary,text,auto (default binary)
	CR/LF -> LF conversion, if auto, decision is made according to extension
	- there is a list of text extensions (I thing it's better to not convert
	text file than to damage binary file). If you want to change that list,
	change it in the source. Original readonly HPFS contained some strange
	heuristic algorithm that I removed. I thing it's danger to let the
	computer decide whether file is text or binary. For example, DJGPP
	binaries contain small text message at the beginning and they could be
	misidentified and damaged under some circumstances.
check=none,normal,strict (default normal)
	Check level. Selecting none will cause only little speedup and big
	danger. I tried to write it so that it won't crash if check=normal on
	corrupted filesystems. check=strict means many superfluous checks -
	used for debugging (for example it checks if file is allocated in
	bitmaps when accessing it).
errors=continue,remount-ro,panic (default remount-ro)
	Behaviour when filesystem errors found.
chkdsk=no,errors,always (default errors)
	When to mark filesystem dirty so that OS/2 checks it.
eas=no,ro,rw (default rw)
	What to do with extended attributes. 'no' - ignore them and use always
	values specified in uid/gid/mode options. 'ro' - read extended
	attributes but do not create them. 'rw' - create extended attributes
	when you use chmod/chown/chgrp/mknod/ln -s on the filesystem.
timeshift=(-)nnn (default 0)
	Shifts the time by nnn seconds. For example, if you see under linux
	one hour more, than under os/2, use timeshift=-3600.

File names
==========

As in OS/2, filenames are case insensitive. However, shell thinks that names
are case sensitive, so for example when you create a file FOO, you can use
'cat FOO', 'cat Foo', 'cat foo' or 'cat F*' but not 'cat f*'. Note, that you
also won't be able to compile linux kernel (and maybe other things) on HPFS
because kernel creates different files with names like bootsect.S and
bootsect.s. When searching for file thats name has characters >= 128, codepages
are used - see below.
OS/2 ignores dots and spaces at the end of file name, so this driver does as
well. If you create 'a. ...', the file 'a' will be created, but you can still
access it under names 'a.', 'a..', 'a .  . . ' etc.

Extended attributes
===================

On HPFS partitions, OS/2 can associate to each file a special information called
extended attributes. Extended attributes are pairs of (key,value) where key is
an ascii string identifying that attribute and value is any string of bytes of
variable length. OS/2 stores window and icon positions and file types there. So
why not use it for unix-specific info like file owner or access rights? This
driver can do it. If you chown/chgrp/chmod on a hpfs partition, extended
attributes with keys "UID", "GID" or "MODE" and 2-byte values are created. Only
that extended attributes those value differs from defaults specified in mount
options are created. Once created, the extended attributes are never deleted,
they're just changed. It means that when your default uid=0 and you type
something like 'chown luser file; chown root file' the file will contain
extended attribute UID=0. And when you umount the fs and mount it again with
uid=luser_uid, the file will be still owned by root! If you chmod file to 444,
extended attribute "MODE" will not be set, this special case is done by setting
read-only flag. When you mknod a block or char device, besides "MODE", the
special 4-byte extended attribute "DEV" will be created containing the device
number. Currently this driver cannot resize extended attributes - it means
that if somebody (I don't know who?) has set "UID", "GID", "MODE" or "DEV"
attributes with different sizes, they won't be rewritten and changing these
values doesn't work.

Symlinks
========

You can do symlinks on HPFS partition, symlinks are achieved by setting extended
attribute named "SYMLINK" with symlink value. Like on ext2, you can chown and
chgrp symlinks but I don't know what is it good for. chmoding symlink results
in chmoding file where symlink points. These symlinks are just for Linux use and
incompatible with OS/2. OS/2 PmShell symlinks are not supported because they are
stored in very crazy way. They tried to do it so that link changes when file is
moved ... sometimes it works. But the link is partly stored in directory
extended attributes and partly in OS2SYS.INI. I don't want (and don't know how)
to analyze or change OS2SYS.INI.

Codepages
=========

HPFS can contain several uppercasing tables for several codepages and each
file has a pointer to codepage its name is in. However OS/2 was created in
America where people don't care much about codepages and so multiple codepages
support is quite buggy. I have Czech OS/2 working in codepage 852 on my disk.
Once I booted English OS/2 working in cp 850 and I created a file on my 852
partition. It marked file name codepage as 850 - good. But when I again booted
Czech OS/2, the file was completely inaccessible under any name. It seems that
OS/2 uppercases the search pattern with its system code page (852) and file
name it's comparing to with its code page (850). These could never match. Is it
really what IBM developers wanted? But problems continued. When I created in
Czech OS/2 another file in that directory, that file was inaccessible too. OS/2
probably uses different uppercasing method when searching where to place a file
(note, that files in HPFS directory must be sorted) and when searching for
a file. Finally when I opened this directory in PmShell, PmShell crashed (the
funny thing was that, when rebooted, PmShell tried to reopen this directory
again :-). chkdsk happily ignores these errors and only low-level disk
modification saved me.  Never mix different language versions of OS/2 on one
system although HPFS was designed to allow that.
OK, I could implement complex codepage support to this driver but I think it
would cause more problems than benefit with such buggy implementation in OS/2.
So this driver simply uses first codepage it finds for uppercasing and
lowercasing no matter what's file codepage index. Usually all file names are in
this codepage - if you don't try to do what I described above :-)

Known bugs
==========

HPFS386 on OS/2 server is not supported. HPFS386 installed on normal OS/2 client
should work. If you have OS/2 server, use only read-only mode. I don't know how
to handle some HPFS386 structures like access control list or extended perm
list, I don't know how to delete them when file is deleted and how to not
overwrite them with extended attributes. Send me some info on these structures
and I'll make it. However, this driver should detect presence of HPFS386
structures, remount read-only and not destroy them (I hope).

When there's not enough space for extended attributes, they will be truncated
and no error is returned.

OS/2 can't access files if the path is longer than about 256 chars but this
driver allows you to do it. chkdsk ignores such errors.

Sometimes you won't be able to delete some files on a very full filesystem
(returning error ENOSPC). That's because file in non-leaf node in directory tree
(one directory, if it's large, has dirents in tree on HPFS) must be replaced
with another node when deleted. And that new file might have larger name than
the old one so the new name doesn't fit in directory node (dnode). And that
would result in directory tree splitting, that takes disk space. Workaround is
to delete other files that are leaf (probability that the file is non-leaf is
about 1/50) or to truncate file first to make some space.
You encounter this problem only if you have many directories so that
preallocated directory band is full i.e.::

	number_of_directories / size_of_filesystem_in_mb > 4.

You can't delete open directories.

You can't rename over directories (what is it good for?).

Renaming files so that only case changes doesn't work. This driver supports it
but vfs doesn't. Something like 'mv file FILE' won't work.

All atimes and directory mtimes are not updated. That's because of performance
reasons. If you extremely wish to update them, let me know, I'll write it (but
it will be slow).

When the system is out of memory and swap, it may slightly corrupt filesystem
(lost files, unbalanced directories). (I guess all filesystem may do it).

When compiled, you get warning: function declaration isn't a prototype. Does
anybody know what does it mean?

What does "unbalanced tree" message mean?
=========================================

Old versions of this driver created sometimes unbalanced dnode trees. OS/2
chkdsk doesn't scream if the tree is unbalanced (and sometimes creates
unbalanced trees too :-) but both HPFS and HPFS386 contain bug that it rarely
crashes when the tree is not balanced. This driver handles unbalanced trees
correctly and writes warning if it finds them. If you see this message, this is
probably because of directories created with old version of this driver.
Workaround is to move all files from that directory to another and then back
again. Do it in Linux, not OS/2! If you see this message in directory that is
whole created by this driver, it is BUG - let me know about it.

Bugs in OS/2
============

When you have two (or more) lost directories pointing each to other, chkdsk
locks up when repairing filesystem.

Sometimes (I think it's random) when you create a file with one-char name under
OS/2, OS/2 marks it as 'long'. chkdsk then removes this flag saying "Minor fs
error corrected".

File names like "a .b" are marked as 'long' by OS/2 but chkdsk "corrects" it and
marks them as short (and writes "minor fs error corrected"). This bug is not in
HPFS386.

Codepage bugs described above
=============================

If you don't install fixpacks, there are many, many more...

History
=======

====== =========================================================================
0.90   First public release
0.91   Fixed bug that caused shooting to memory when write_inode was called on
       open inode (rarely happened)
0.92   Fixed a little memory leak in freeing directory inodes
0.93   Fixed bug that locked up the machine when there were too many filenames
       with first 15 characters same
       Fixed write_file to zero file when writing behind file end
0.94   Fixed a little memory leak when trying to delete busy file or directory
0.95   Fixed a bug that i_hpfs_parent_dir was not updated when moving files
1.90   First version for 2.1.1xx kernels
1.91   Fixed a bug that chk_sectors failed when sectors were at the end of disk
       Fixed a race-condition when write_inode is called while deleting file
       Fixed a bug that could possibly happen (with very low probability) when
       using 0xff in filenames.

       Rewritten locking to avoid race-conditions

       Mount option 'eas' now works

       Fsync no longer returns error

       Files beginning with '.' are marked hidden

       Remount support added

       Alloc is not so slow when filesystem becomes full

       Atimes are no more updated because it slows down operation

       Code cleanup (removed all commented debug prints)
1.92   Corrected a bug when sync was called just before closing file
1.93   Modified, so that it works with kernels >= 2.1.131, I don't know if it
       works with previous versions

       Fixed a possible problem with disks > 64G (but I don't have one, so I can't
       test it)

       Fixed a file overflow at 2G

       Added new option 'timeshift'

       Changed behaviour on HPFS386: It is now possible to operate on HPFS386 in
       read-only mode

       Fixed a bug that slowed down alloc and prevented allocating 100% space
       (this bug was not destructive)
1.94   Added workaround for one bug in Linux

       Fixed one buffer leak

       Fixed some incompatibilities with large extended attributes (but it's still
       not 100% ok, I have no info on it and OS/2 doesn't want to create them)

       Rewritten allocation

       Fixed a bug with i_blocks (du sometimes didn't display correct values)

       Directories have no longer archive attribute set (some programs don't like
       it)

       Fixed a bug that it set badly one flag in large anode tree (it was not
       destructive)
1.95   Fixed one buffer leak, that could happen on corrupted filesystem

       Fixed one bug in allocation in 1.94
1.96   Added workaround for one bug in OS/2 (HPFS locked up, HPFS386 reported
       error sometimes when opening directories in PMSHELL)

       Fixed a possible bitmap race

       Fixed possible problem on large disks

       You can now delete open files

       Fixed a nondestructive race in rename
1.97   Support for HPFS v3 (on large partitions)

       ZFixed a bug that it didn't allow creation of files > 128M
       (it should be 2G)
1.97.1 Changed names of global symbols

       Fixed a bug when chmoding or chowning root directory
1.98   Fixed a deadlock when using old_readdir
       Better directory handling; workaround for "unbalanced tree" bug in OS/2
1.99   Corrected a possible problem when there's not enough space while deleting
       file

       Now it tries to truncate the file if there's not enough space when
       deleting

       Removed a lot of redundant code
2.00   Fixed a bug in rename (it was there since 1.96)
       Better anti-fragmentation strategy
2.01   Fixed problem with directory listing over NFS

       Directory lseek now checks for proper parameters

       Fixed race-condition in buffer code - it is in all filesystems in Linux;
       when reading device (cat /dev/hda) while creating files on it, files
       could be damaged
2.02   Workaround for bug in breada in Linux. breada could cause accesses beyond
       end of partition
2.03   Char, block devices and pipes are correctly created

       Fixed non-crashing race in unlink (Alexander Viro)

       Now it works with Japanese version of OS/2
2.04   Fixed error when ftruncate used to extend file
2.05   Fixed crash when got mount parameters without =

       Fixed crash when allocation of anode failed due to full disk

       Fixed some crashes when block io or inode allocation failed
2.06   Fixed some crash on corrupted disk structures

       Better allocation strategy

       Reschedule points added so that it doesn't lock CPU long time

       It should work in read-only mode on Warp Server
2.07   More fixes for Warp Server. Now it really works
2.08   Creating new files is not so slow on large disks

       An attempt to sync deleted file does not generate filesystem error
2.09   Fixed error on extremely fragmented files
====== =========================================================================