code:
.. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0)
.. See the bottom of this file for additional redistribution information.

Handling regressions
++++++++++++++++++++

*We don't cause regressions* -- this document describes what this "first rule of
Linux kernel development" means in practice for developers. It complements
Documentation/admin-guide/reporting-regressions.rst, which covers the topic from a
user's point of view; if you never read that text, go and at least skim over it
before continuing here.

The important bits (aka "The TL;DR")
====================================

#. Ensure subscribers of the `regression mailing list <https://lore.kernel.org/regressions/>`_
   (regressions@lists.linux.dev) quickly become aware of any new regression
   report:

    * When receiving a mailed report that did not CC the list, bring it into the
      loop by immediately sending at least a brief "Reply-all" with the list
      CCed.

    * Forward or bounce any reports submitted in bug trackers to the list.

#. Make the Linux kernel regression tracking bot "regzbot" track the issue (this
   is optional, but recommended):

    * For mailed reports, check if the reporter included a line like ``#regzbot
      introduced v5.13..v5.14-rc1``. If not, send a reply (with the regressions
      list in CC) containing a paragraph like the following, which tells regzbot
      when the issue started to happen::

       #regzbot ^introduced 1f2e3d4c5b6a

    * When forwarding reports from a bug tracker to the regressions list (see
      above), include a paragraph like the following::

       #regzbot introduced: v5.13..v5.14-rc1
       #regzbot from: Some N. Ice Human <some.human@example.com>
       #regzbot monitor: http://some.bugtracker.example.com/ticket?id=123456789

#. When submitting fixes for regressions, add "Link:" tags to the patch
   description pointing to all places where the issue was reported, as
   mandated by Documentation/process/submitting-patches.rst and
   :ref:`Documentation/process/5.Posting.rst <development_posting>`.

#. Try to fix regressions quickly once the culprit has been identified; fixes
   for most regressions should be merged within two weeks, but some need to be
   resolved within two or three days.

All the details on Linux kernel regressions relevant for developers
===================================================================

The important basics in more detail
-----------------------------------

What to do when receiving regression reports
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Ensure the Linux kernel's regression tracker and others subscribers of the
`regression mailing list <https://lore.kernel.org/regressions/>`_
(regressions@lists.linux.dev) become aware of any newly reported regression:

 * When you receive a report by mail that did not CC the list, immediately bring
   it into the loop by sending at least a brief "Reply-all" with the list CCed;
   try to ensure it gets CCed again in case you reply to a reply that omitted
   the list.

 * If a report submitted in a bug tracker hits your Inbox, forward or bounce it
   to the list. Consider checking the list archives beforehand, if the reporter
   already forwarded the report as instructed by
   Documentation/admin-guide/reporting-issues.rst.

When doing either, consider making the Linux kernel regression tracking bot
"regzbot" immediately start tracking the issue:

 * For mailed reports, check if the reporter included a "regzbot command" like
   ``#regzbot introduced 1f2e3d4c5b6a``. If not, send a reply (with the
   regressions list in CC) with a paragraph like the following:::

       #regzbot ^introduced: v5.13..v5.14-rc1

   This tells regzbot the version range in which the issue started to happen;
   you can specify a range using commit-ids as well or state a single commit-id
   in case the reporter bisected the culprit.

   Note the caret (^) before the "introduced": it tells regzbot to treat the
   parent mail (the one you reply to) as the initial report for the regression
   you want to see tracked; that's important, as regzbot will later look out
   for patches with "Link:" tags pointing to the report in the archives on
   lore.kernel.org.

 * When forwarding a regressions reported to a bug tracker, include a paragraph
   with these regzbot commands::

       #regzbot introduced: 1f2e3d4c5b6a
       #regzbot from: Some N. Ice Human <some.human@example.com>
       #regzbot monitor: http://some.bugtracker.example.com/ticket?id=123456789

   Regzbot will then automatically associate patches with the report that
   contain "Link:" tags pointing to your mail or the mentioned ticket.

What's important when fixing regressions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You don't need to do anything special when submitting fixes for regression, just
remember to do what Documentation/process/submitting-patches.rst,
:ref:`Documentation/process/5.Posting.rst <development_posting>`, and
Documentation/process/stable-kernel-rules.rst already explain in more detail:

 * Point to all places where the issue was reported using "Link:" tags::

       Link: https://lore.kernel.org/r/30th.anniversary.repost@klaava.Helsinki.FI/
       Link: https://bugzilla.kernel.org/show_bug.cgi?id=1234567890

 * Add a "Fixes:" tag to specify the commit causing the regression.

 * If the culprit was merged in an earlier development cycle, explicitly mark
   the fix for backporting using the ``Cc: stable@vger.kernel.org`` tag.

All this is expected from you and important when it comes to regression, as
these tags are of great value for everyone (you included) that might be looking
into the issue weeks, months, or years later. These tags are also crucial for
tools and scripts used by other kernel developers or Linux distributions; one of
these tools is regzbot, which heavily relies on the "Link:" tags to associate
reports for regression with changes resolving them.

Prioritize work on fixing regressions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You should fix any reported regression as quickly as possible, to provide
affected users with a solution in a timely manner and prevent more users from
running into the issue; nevertheless developers need to take enough time and
care to ensure regression fixes do not cause additional damage.

In the end though, developers should give their best to prevent users from
running into situations where a regression leaves them only three options: "run
a kernel with a regression that seriously impacts usage", "continue running an
outdated and thus potentially insecure kernel version for more than two weeks
after a regression's culprit was identified", and "downgrade to a still
supported kernel series that lack required features".

How to realize this depends a lot on the situation. Here are a few rules of
thumb for you, in order or importance:

 * Prioritize work on handling regression reports and fixing regression over all
   other Linux kernel work, unless the latter concerns acute security issues or
   bugs causing data loss or damage.

 * Always consider reverting the culprit commits and reapplying them later
   together with necessary fixes, as this might be the least dangerous and
   quickest way to fix a regression.

 * Developers should handle regressions in all supported kernel series, but are
   free to delegate the work to the stable team, if the issue probably at no
   point in time occurred with mainline.

 * Try to resolve any regressions introduced in the current development before
   its end. If you fear a fix might be too risky to apply only days before a new
   mainline release, let Linus decide: submit the fix separately to him as soon
   as possible with the explanation of the situation. He then can make a call
   and postpone the release if necessary, for example if multiple such changes
   show up in his inbox.

 * Address regressions in stable, longterm, or proper mainline releases with
   more urgency than regressions in mainline pre-releases. That changes after
   the release of the fifth pre-release, aka "-rc5": mainline then becomes as
   important, to ensure all the improvements and fixes are ideally tested
   together for at least one week before Linus releases a new mainline version.

 * Fix regressions within two or three days, if they are critical for some
   reason -- for example, if the issue is likely to affect many users of the
   kernel series in question on all or certain architectures. Note, this
   includes mainline, as issues like compile errors otherwise might prevent many
   testers or continuous integration systems from testing the series.

 * Aim to fix regressions within one week after the culprit was identified, if
   the issue was introduced in either:

    * a recent stable/longterm release

    * the development cycle of the latest proper mainline release

   In the latter case (say Linux v5.14), try to address regressions even
   quicker, if the stable series for the predecessor (v5.13) will be abandoned
   soon or already was stamped "End-of-Life" (EOL) -- this usually happens about
   three to four weeks after a new mainline release.

 * Try to fix all other regressions within two weeks after the culprit was
   found. Two or three additional weeks are acceptable for performance
   regressions and other issues which are annoying, but don't prevent anyone
   from running Linux (unless it's an issue in the current development cycle,
   as those should ideally be addressed before the release). A few weeks in
   total are acceptable if a regression can only be fixed with a risky change
   and at the same time is affecting only a few users; as much time is
   also okay if the regression is already present in the second newest longterm
   kernel series.

Note: The aforementioned time frames for resolving regressions are meant to
include getting the fix tested, reviewed, and merged into mainline, ideally with
the fix being in linux-next at least briefly. This leads to delays you need to
account for.

Subsystem maintainers are expected to assist in reaching those periods by doing
timely reviews and quick handling of accepted patches. They thus might have to
send git-pull requests earlier or more often than usual; depending on the fix,
it might even be acceptable to skip testing in linux-next. Especially fixes for
regressions in stable and longterm kernels need to be handled quickly, as fixes
need to be merged in mainline before they can be backported to older series.

More aspects regarding regressions developers should be aware of
----------------------------------------------------------------

How to deal with changes where a risk of regression is known
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Evaluate how big the risk of regressions is, for example by performing a code
search in Linux distributions and Git forges. Also consider asking other
developers or projects likely to be affected to evaluate or even test the
proposed change; if problems surface, maybe some solution acceptable for all
can be found.

If the risk of regressions in the end seems to be relatively small, go ahead
with the change, but let all involved parties know about the risk. Hence, make
sure your patch description makes this aspect obvious. Once the change is
merged, tell the Linux kernel's regression tracker and the regressions mailing
list about the risk, so everyone has the change on the radar in case reports
trickle in. Depending on the risk, you also might want to ask the subsystem
maintainer to mention the issue in his mainline pull request.

What else is there to known about regressions?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Check out Documentation/admin-guide/reporting-regressions.rst, it covers a lot
of other aspects you want might want to be aware of:

 * the purpose of the "no regressions rule"

 * what issues actually qualify as regression

 * who's in charge for finding the root cause of a regression

 * how to handle tricky situations, e.g. when a regression is caused by a
   security fix or when fixing a regression might cause another one

Whom to ask for advice when it comes to regressions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Send a mail to the regressions mailing list (regressions@lists.linux.dev) while
CCing the Linux kernel's regression tracker (regressions@leemhuis.info); if the
issue might better be dealt with in private, feel free to omit the list.

More about regression tracking and regzbot
------------------------------------------

Why the Linux kernel has a regression tracker, and why is regzbot used?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Rules like "no regressions" need someone to ensure they are followed, otherwise
they are broken either accidentally or on purpose. History has shown this to be
true for the Linux kernel as well. That's why Thorsten Leemhuis volunteered to
keep an eye on things as the Linux kernel's regression tracker, who's
occasionally helped by other people. Neither of them are paid to do this,
that's why regression tracking is done on a best effort basis.

Earlier attempts to manually track regressions have shown it's an exhausting and
frustrating work, which is why they were abandoned after a while. To prevent
this from happening again, Thorsten developed regzbot to facilitate the work,
with the long term goal to automate regression tracking as much as possible for
everyone involved.

How does regression tracking work with regzbot?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The bot watches for replies to reports of tracked regressions. Additionally,
it's looking out for posted or committed patches referencing such reports
with "Link:" tags; replies to such patch postings are tracked as well.
Combined this data provides good insights into the current state of the fixing
process.

Regzbot tries to do its job with as little overhead as possible for both
reporters and developers. In fact, only reporters are burdened with an extra
duty: they need to tell regzbot about the regression report using the ``#regzbot
introduced`` command outlined above; if they don't do that, someone else can
take care of that using ``#regzbot ^introduced``.

For developers there normally is no extra work involved, they just need to make
sure to do something that was expected long before regzbot came to light: add
"Link:" tags to the patch description pointing to all reports about the issue
fixed.

Do I have to use regzbot?
~~~~~~~~~~~~~~~~~~~~~~~~~

It's in the interest of everyone if you do, as kernel maintainers like Linus
Torvalds partly rely on regzbot's tracking in their work -- for example when
deciding to release a new version or extend the development phase. For this they
need to be aware of all unfixed regression; to do that, Linus is known to look
into the weekly reports sent by regzbot.

Do I have to tell regzbot about every regression I stumble upon?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Ideally yes: we are all humans and easily forget problems when something more
important unexpectedly comes up -- for example a bigger problem in the Linux
kernel or something in real life that's keeping us away from keyboards for a
while. Hence, it's best to tell regzbot about every regression, except when you
immediately write a fix and commit it to a tree regularly merged to the affected
kernel series.

How to see which regressions regzbot tracks currently?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Check `regzbot's web-interface <https://linux-regtracking.leemhuis.info/regzbot/>`_
for the latest info; alternatively, `search for the latest regression report
<https://lore.kernel.org/lkml/?q=%22Linux+regressions+report%22+f%3Aregzbot>`_,
which regzbot normally sends out once a week on Sunday evening (UTC), which is a
few hours before Linus usually publishes new (pre-)releases.

What places is regzbot monitoring?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Regzbot is watching the most important Linux mailing lists as well as the git
repositories of linux-next, mainline, and stable/longterm.

What kind of issues are supposed to be tracked by regzbot?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The bot is meant to track regressions, hence please don't involve regzbot for
regular issues. But it's okay for the Linux kernel's regression tracker if you
use regzbot to track severe issues, like reports about hangs, corrupted data,
or internal errors (Panic, Oops, BUG(), warning, ...).

Can I add regressions found by CI systems to regzbot's tracking?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Feel free to do so, if the particular regression likely has impact on practical
use cases and thus might be noticed by users; hence, please don't involve
regzbot for theoretical regressions unlikely to show themselves in real world
usage.

How to interact with regzbot?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

By using a 'regzbot command' in a direct or indirect reply to the mail with the
regression report. These commands need to be in their own paragraph (IOW: they
need to be separated from the rest of the mail using blank lines).

One such command is ``#regzbot introduced <version or commit>``, which makes
regzbot consider your mail as a regressions report added to the tracking, as
already described above; ``#regzbot ^introduced <version or commit>`` is another
such command, which makes regzbot consider the parent mail as a report for a
regression which it starts to track.

Once one of those two commands has been utilized, other regzbot commands can be
used in direct or indirect replies to the report. You can write them below one
of the `introduced` commands or in replies to the mail that used one of them
or itself is a reply to that mail:

 * Set or update the title::

       #regzbot title: foo

 * Monitor a discussion or bugzilla.kernel.org ticket where additions aspects of
   the issue or a fix are discussed -- for example the posting of a patch fixing
   the regression::

       #regzbot monitor: https://lore.kernel.org/all/30th.anniversary.repost@klaava.Helsinki.FI/

   Monitoring only works for lore.kernel.org and bugzilla.kernel.org; regzbot
   will consider all messages in that thread or ticket as related to the fixing
   process.

 * Point to a place with further details of interest, like a mailing list post
   or a ticket in a bug tracker that are slightly related, but about a different
   topic::

       #regzbot link: https://bugzilla.kernel.org/show_bug.cgi?id=123456789

 * Mark a regression as fixed by a commit that is heading upstream or already
   landed::

       #regzbot fixed-by: 1f2e3d4c5d

 * Mark a regression as a duplicate of another one already tracked by regzbot::

       #regzbot dup-of: https://lore.kernel.org/all/30th.anniversary.repost@klaava.Helsinki.FI/

 * Mark a regression as invalid::

       #regzbot invalid: wasn't a regression, problem has always existed

Is there more to tell about regzbot and its commands?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

More detailed and up-to-date information about the Linux
kernel's regression tracking bot can be found on its
`project page <https://gitlab.com/knurd42/regzbot>`_, which among others
contains a `getting started guide <https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md>`_
and `reference documentation <https://gitlab.com/knurd42/regzbot/-/blob/main/docs/reference.md>`_
which both cover more details than the above section.

Quotes from Linus about regression
----------------------------------

Find below a few real life examples of how Linus Torvalds expects regressions to
be handled:

 * From `2017-10-26 (1/2)
   <https://lore.kernel.org/lkml/CA+55aFwiiQYJ+YoLKCXjN_beDVfu38mg=Ggg5LFOcqHE8Qi7Zw@mail.gmail.com/>`_::

       If you break existing user space setups THAT IS A REGRESSION.

       It's not ok to say "but we'll fix the user space setup".

       Really. NOT OK.

       [...]

       The first rule is:

        - we don't cause regressions

       and the corollary is that when regressions *do* occur, we admit to
       them and fix them, instead of blaming user space.

       The fact that you have apparently been denying the regression now for
       three weeks means that I will revert, and I will stop pulling apparmor
       requests until the people involved understand how kernel development
       is done.

 * From `2017-10-26 (2/2)
   <https://lore.kernel.org/lkml/CA+55aFxW7NMAMvYhkvz1UPbUTUJewRt6Yb51QAx5RtrWOwjebg@mail.gmail.com/>`_::

       People should basically always feel like they can update their kernel
       and simply not have to worry about it.

       I refuse to introduce "you can only update the kernel if you also
       update that other program" kind of limitations. If the kernel used to
       work for you, the rule is that it continues to work for you.

       There have been exceptions, but they are few and far between, and they
       generally have some major and fundamental reasons for having happened,
       that were basically entirely unavoidable, and people _tried_hard_ to
       avoid them. Maybe we can't practically support the hardware any more
       after it is decades old and nobody uses it with modern kernels any
       more. Maybe there's a serious security issue with how we did things,
       and people actually depended on that fundamentally broken model. Maybe
       there was some fundamental other breakage that just _had_ to have a
       flag day for very core and fundamental reasons.

       And notice that this is very much about *breaking* peoples environments.

       Behavioral changes happen, and maybe we don't even support some
       feature any more. There's a number of fields in /proc/<pid>/stat that
       are printed out as zeroes, simply because they don't even *exist* in
       the kernel any more, or because showing them was a mistake (typically
       an information leak). But the numbers got replaced by zeroes, so that
       the code that used to parse the fields still works. The user might not
       see everything they used to see, and so behavior is clearly different,
       but things still _work_, even if they might no longer show sensitive
       (or no longer relevant) information.

       But if something actually breaks, then the change must get fixed or
       reverted. And it gets fixed in the *kernel*. Not by saying "well, fix
       your user space then". It was a kernel change that exposed the
       problem, it needs to be the kernel that corrects for it, because we
       have a "upgrade in place" model. We don't have a "upgrade with new
       user space".

       And I seriously will refuse to take code from people who do not
       understand and honor this very simple rule.

       This rule is also not going to change.

       And yes, I realize that the kernel is "special" in this respect. I'm
       proud of it.

       I have seen, and can point to, lots of projects that go "We need to
       break that use case in order to make progress" or "you relied on
       undocumented behavior, it sucks to be you" or "there's a better way to
       do what you want to do, and you have to change to that new better
       way", and I simply don't think that's acceptable outside of very early
       alpha releases that have experimental users that know what they signed
       up for. The kernel hasn't been in that situation for the last two
       decades.

       We do API breakage _inside_ the kernel all the time. We will fix
       internal problems by saying "you now need to do XYZ", but then it's
       about internal kernel API's, and the people who do that then also
       obviously have to fix up all the in-kernel users of that API. Nobody
       can say "I now broke the API you used, and now _you_ need to fix it
       up". Whoever broke something gets to fix it too.

       And we simply do not break user space.

 * From `2020-05-21
   <https://lore.kernel.org/all/CAHk-=wiVi7mSrsMP=fLXQrXK_UimybW=ziLOwSzFTtoXUacWVQ@mail.gmail.com/>`_::

       The rules about regressions have never been about any kind of
       documented behavior, or where the code lives.

       The rules about regressions are always about "breaks user workflow".

       Users are literally the _only_ thing that matters.

       No amount of "you shouldn't have used this" or "that behavior was
       undefined, it's your own fault your app broke" or "that used to work
       simply because of a kernel bug" is at all relevant.

       Now, reality is never entirely black-and-white. So we've had things
       like "serious security issue" etc that just forces us to make changes
       that may break user space. But even then the rule is that we don't
       really have other options that would allow things to continue.

       And obviously, if users take years to even notice that something
       broke, or if we have sane ways to work around the breakage that
       doesn't make for too much trouble for users (ie "ok, there are a
       handful of users, and they can use a kernel command line to work
       around it" kind of things) we've also been a bit less strict.

       But no, "that was documented to be broken" (whether it's because the
       code was in staging or because the man-page said something else) is
       irrelevant. If staging code is so useful that people end up using it,
       that means that it's basically regular kernel code with a flag saying
       "please clean this up".

       The other side of the coin is that people who talk about "API
       stability" are entirely wrong. API's don't matter either. You can make
       any changes to an API you like - as long as nobody notices.

       Again, the regression rule is not about documentation, not about
       API's, and not about the phase of the moon.

       It's entirely about "we caused problems for user space that used to work".

 * From `2017-11-05
   <https://lore.kernel.org/all/CA+55aFzUvbGjD8nQ-+3oiMBx14c_6zOj2n7KLN3UsJ-qsd4Dcw@mail.gmail.com/>`_::

       And our regression rule has never been "behavior doesn't change".
       That would mean that we could never make any changes at all.

       For example, we do things like add new error handling etc all the
       time, which we then sometimes even add tests for in our kselftest
       directory.

       So clearly behavior changes all the time and we don't consider that a
       regression per se.

       The rule for a regression for the kernel is that some real user
       workflow breaks. Not some test. Not a "look, I used to be able to do
       X, now I can't".

 * From `2018-08-03
   <https://lore.kernel.org/all/CA+55aFwWZX=CXmWDTkDGb36kf12XmTehmQjbiMPCqCRG2hi9kw@mail.gmail.com/>`_::

       YOU ARE MISSING THE #1 KERNEL RULE.

       We do not regress, and we do not regress exactly because your are 100% wrong.

       And the reason you state for your opinion is in fact exactly *WHY* you
       are wrong.

       Your "good reasons" are pure and utter garbage.

       The whole point of "we do not regress" is so that people can upgrade
       the kernel and never have to worry about it.

       > Kernel had a bug which has been fixed

       That is *ENTIRELY* immaterial.

       Guys, whether something was buggy or not DOES NOT MATTER.

       Why?

       Bugs happen. That's a fact of life. Arguing that "we had to break
       something because we were fixing a bug" is completely insane. We fix
       tens of bugs every single day, thinking that "fixing a bug" means that
       we can break something is simply NOT TRUE.

       So bugs simply aren't even relevant to the discussion. They happen,
       they get found, they get fixed, and it has nothing to do with "we
       break users".

       Because the only thing that matters IS THE USER.

       How hard is that to understand?

       Anybody who uses "but it was buggy" as an argument is entirely missing
       the point. As far as the USER was concerned, it wasn't buggy - it
       worked for him/her.

       Maybe it worked *because* the user had taken the bug into account,
       maybe it worked because the user didn't notice - again, it doesn't
       matter. It worked for the user.

       Breaking a user workflow for a "bug" is absolutely the WORST reason
       for breakage you can imagine.

       It's basically saying "I took something that worked, and I broke it,
       but now it's better". Do you not see how f*cking insane that statement
       is?

       And without users, your program is not a program, it's a pointless
       piece of code that you might as well throw away.

       Seriously. This is *why* the #1 rule for kernel development is "we
       don't break users". Because "I fixed a bug" is absolutely NOT AN
       ARGUMENT if that bug fix broke a user setup. You actually introduced a
       MUCH BIGGER bug by "fixing" something that the user clearly didn't
       even care about.

       And dammit, we upgrade the kernel ALL THE TIME without upgrading any
       other programs at all. It is absolutely required, because flag-days
       and dependencies are horribly bad.

       And it is also required simply because I as a kernel developer do not
       upgrade random other tools that I don't even care about as I develop
       the kernel, and I want any of my users to feel safe doing the same
       time.

       So no. Your rule is COMPLETELY wrong. If you cannot upgrade a kernel
       without upgrading some other random binary, then we have a problem.

 * From `2021-06-05
   <https://lore.kernel.org/all/CAHk-=wiUVqHN76YUwhkjZzwTdjMMJf_zN4+u7vEJjmEGh3recw@mail.gmail.com/>`_::

       THERE ARE NO VALID ARGUMENTS FOR REGRESSIONS.

       Honestly, security people need to understand that "not working" is not
       a success case of security. It's a failure case.

       Yes, "not working" may be secure. But security in that case is *pointless*.

 * From `2011-05-06 (1/3)
   <https://lore.kernel.org/all/BANLkTim9YvResB+PwRp7QTK-a5VNg2PvmQ@mail.gmail.com/>`_::

       Binary compatibility is more important.

       And if binaries don't use the interface to parse the format (or just
       parse it wrongly - see the fairly recent example of adding uuid's to
       /proc/self/mountinfo), then it's a regression.

       And regressions get reverted, unless there are security issues or
       similar that makes us go "Oh Gods, we really have to break things".

       I don't understand why this simple logic is so hard for some kernel
       developers to understand. Reality matters. Your personal wishes matter
       NOT AT ALL.

       If you made an interface that can be used without parsing the
       interface description, then we're stuck with the interface. Theory
       simply doesn't matter.

       You could help fix the tools, and try to avoid the compatibility
       issues that way. There aren't that many of them.

   From `2011-05-06 (2/3)
   <https://lore.kernel.org/all/BANLkTi=KVXjKR82sqsz4gwjr+E0vtqCmvA@mail.gmail.com/>`_::

       it's clearly NOT an internal tracepoint. By definition. It's being
       used by powertop.

   From `2011-05-06 (3/3)
   <https://lore.kernel.org/all/BANLkTinazaXRdGovYL7rRVp+j6HbJ7pzhg@mail.gmail.com/>`_::

       We have programs that use that ABI and thus it's a regression if they break.

 * From `2012-07-06 <https://lore.kernel.org/all/CA+55aFwnLJ+0sjx92EGREGTWOx84wwKaraSzpTNJwPVV8edw8g@mail.gmail.com/>`_::

       > Now this got me wondering if Debian _unstable_ actually qualifies as a
       > standard distro userspace.

       Oh, if the kernel breaks some standard user space, that counts. Tons
       of people run Debian unstable

 * From `2019-09-15
   <https://lore.kernel.org/lkml/CAHk-=wiP4K8DRJWsCo=20hn_6054xBamGKF2kPgUzpB5aMaofA@mail.gmail.com/>`_::

       One _particularly_ last-minute revert is the top-most commit (ignoring
       the version change itself) done just before the release, and while
       it's very annoying, it's perhaps also instructive.

       What's instructive about it is that I reverted a commit that wasn't
       actually buggy. In fact, it was doing exactly what it set out to do,
       and did it very well. In fact it did it _so_ well that the much
       improved IO patterns it caused then ended up revealing a user-visible
       regression due to a real bug in a completely unrelated area.

       The actual details of that regression are not the reason I point that
       revert out as instructive, though. It's more that it's an instructive
       example of what counts as a regression, and what the whole "no
       regressions" kernel rule means. The reverted commit didn't change any
       API's, and it didn't introduce any new bugs. But it ended up exposing
       another problem, and as such caused a kernel upgrade to fail for a
       user. So it got reverted.

       The point here being that we revert based on user-reported _behavior_,
       not based on some "it changes the ABI" or "it caused a bug" concept.
       The problem was really pre-existing, and it just didn't happen to
       trigger before. The better IO patterns introduced by the change just
       happened to expose an old bug, and people had grown to depend on the
       previously benign behavior of that old issue.

       And never fear, we'll re-introduce the fix that improved on the IO
       patterns once we've decided just how to handle the fact that we had a
       bad interaction with an interface that people had then just happened
       to rely on incidental behavior for before. It's just that we'll have
       to hash through how to do that (there are no less than three different
       patches by three different developers being discussed, and there might
       be more coming...). In the meantime, I reverted the thing that exposed
       the problem to users for this release, even if I hope it will be
       re-introduced (perhaps even backported as a stable patch) once we have
       consensus about the issue it exposed.

       Take-away from the whole thing: it's not about whether you change the
       kernel-userspace ABI, or fix a bug, or about whether the old code
       "should never have worked in the first place". It's about whether
       something breaks existing users' workflow.

       Anyway, that was my little aside on the whole regression thing.  Since
       it's that "first rule of kernel programming", I felt it is perhaps
       worth just bringing it up every once in a while

..
   end-of-content
..
   This text is available under GPL-2.0+ or CC-BY-4.0, as stated at the top
   of the file. If you want to distribute this text under CC-BY-4.0 only,
   please use "The Linux kernel developers" for author attribution and link
   this as source:
   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/process/handling-regressions.rst
..
   Note: Only the content of this RST file as found in the Linux kernel sources
   is available under CC-BY-4.0, as versions of this text that were processed
   (for example by the kernel's build system) might contain content taken from
   files which use a more restrictive license.


code:
.. _process_howto:

HOWTO do Linux kernel development
=================================

This is the be-all, end-all document on this topic.  It contains
instructions on how to become a Linux kernel developer and how to learn
to work with the Linux kernel development community.  It tries to not
contain anything related to the technical aspects of kernel programming,
but will help point you in the right direction for that.

If anything in this document becomes out of date, please send in patches
to the maintainer of this file, who is listed at the bottom of the
document.

Introduction
------------

So, you want to learn how to become a Linux kernel developer?  Or you
have been told by your manager, "Go write a Linux driver for this
device."  This document's goal is to teach you everything you need to
know to achieve this by describing the process you need to go through,
and hints on how to work with the community.  It will also try to
explain some of the reasons why the community works like it does.

The kernel is written mostly in C, with some architecture-dependent
parts written in assembly. A good understanding of C is required for
kernel development.  Assembly (any architecture) is not required unless
you plan to do low-level development for that architecture.  Though they
are not a good substitute for a solid C education and/or years of
experience, the following books are good for, if anything, reference:

 - "The C Programming Language" by Kernighan and Ritchie [Prentice Hall]
 - "Practical C Programming" by Steve Oualline [O'Reilly]
 - "C:  A Reference Manual" by Harbison and Steele [Prentice Hall]

The kernel is written using GNU C and the GNU toolchain.  While it
adheres to the ISO C11 standard, it uses a number of extensions that are
not featured in the standard.  The kernel is a freestanding C
environment, with no reliance on the standard C library, so some
portions of the C standard are not supported.  Arbitrary long long
divisions and floating point are not allowed.  It can sometimes be
difficult to understand the assumptions the kernel has on the toolchain
and the extensions that it uses, and unfortunately there is no
definitive reference for them.  Please check the gcc info pages (`info
gcc`) for some information on them.

Please remember that you are trying to learn how to work with the
existing development community.  It is a diverse group of people, with
high standards for coding, style and procedure.  These standards have
been created over time based on what they have found to work best for
such a large and geographically dispersed team.  Try to learn as much as
possible about these standards ahead of time, as they are well
documented; do not expect people to adapt to you or your company's way
of doing things.

Legal Issues
------------

The Linux kernel source code is released under the GPL.  Please see the file
COPYING in the main directory of the source tree. The Linux kernel licensing
rules and how to use `SPDX <https://spdx.org/>`_ identifiers in source code are
described in :ref:`Documentation/process/license-rules.rst <kernel_licensing>`.
If you have further questions about the license, please contact a lawyer, and do
not ask on the Linux kernel mailing list.  The people on the mailing lists are
not lawyers, and you should not rely on their statements on legal matters.

For common questions and answers about the GPL, please see:

	https://www.gnu.org/licenses/gpl-faq.html

Documentation
-------------

The Linux kernel source tree has a large range of documents that are
invaluable for learning how to interact with the kernel community.  When
new features are added to the kernel, it is recommended that new
documentation files are also added which explain how to use the feature.
When a kernel change causes the interface that the kernel exposes to
userspace to change, it is recommended that you send the information or
a patch to the manual pages explaining the change to the manual pages
maintainer at mtk.manpages@gmail.com, and CC the list
linux-api@vger.kernel.org.

Here is a list of files that are in the kernel source tree that are
required reading:

  :ref:`Documentation/admin-guide/README.rst <readme>`
    This file gives a short background on the Linux kernel and describes
    what is necessary to do to configure and build the kernel.  People
    who are new to the kernel should start here.

  :ref:`Documentation/process/changes.rst <changes>`
    This file gives a list of the minimum levels of various software
    packages that are necessary to build and run the kernel
    successfully.

  :ref:`Documentation/process/coding-style.rst <codingstyle>`
    This describes the Linux kernel coding style, and some of the
    rationale behind it. All new code is expected to follow the
    guidelines in this document. Most maintainers will only accept
    patches if these rules are followed, and many people will only
    review code if it is in the proper style.

  :ref:`Documentation/process/submitting-patches.rst <submittingpatches>`
    This file describes in explicit detail how to successfully create
    and send a patch, including (but not limited to):

       - Email contents
       - Email format
       - Who to send it to

    Following these rules will not guarantee success (as all patches are
    subject to scrutiny for content and style), but not following them
    will almost always prevent it.

    Other excellent descriptions of how to create patches properly are:

	"The Perfect Patch"
		https://www.ozlabs.org/~akpm/stuff/tpp.txt

	"Linux kernel patch submission format"
		https://web.archive.org/web/20180829112450/http://linux.yyz.us/patch-format.html

  :ref:`Documentation/process/stable-api-nonsense.rst <stable_api_nonsense>`
    This file describes the rationale behind the conscious decision to
    not have a stable API within the kernel, including things like:

      - Subsystem shim-layers (for compatibility?)
      - Driver portability between Operating Systems.
      - Mitigating rapid change within the kernel source tree (or
	preventing rapid change)

    This document is crucial for understanding the Linux development
    philosophy and is very important for people moving to Linux from
    development on other Operating Systems.

  :ref:`Documentation/admin-guide/security-bugs.rst <securitybugs>`
    If you feel you have found a security problem in the Linux kernel,
    please follow the steps in this document to help notify the kernel
    developers, and help solve the issue.

  :ref:`Documentation/process/management-style.rst <managementstyle>`
    This document describes how Linux kernel maintainers operate and the
    shared ethos behind their methodologies.  This is important reading
    for anyone new to kernel development (or anyone simply curious about
    it), as it resolves a lot of common misconceptions and confusion
    about the unique behavior of kernel maintainers.

  :ref:`Documentation/process/stable-kernel-rules.rst <stable_kernel_rules>`
    This file describes the rules on how the stable kernel releases
    happen, and what to do if you want to get a change into one of these
    releases.

  :ref:`Documentation/process/kernel-docs.rst <kernel_docs>`
    A list of external documentation that pertains to kernel
    development.  Please consult this list if you do not find what you
    are looking for within the in-kernel documentation.

  :ref:`Documentation/process/applying-patches.rst <applying_patches>`
    A good introduction describing exactly what a patch is and how to
    apply it to the different development branches of the kernel.

The kernel also has a large number of documents that can be
automatically generated from the source code itself or from
ReStructuredText markups (ReST), like this one. This includes a
full description of the in-kernel API, and rules on how to handle
locking properly.

All such documents can be generated as PDF or HTML by running::

	make pdfdocs
	make htmldocs

respectively from the main kernel source directory.

The documents that uses ReST markup will be generated at Documentation/output.
They can also be generated on LaTeX and ePub formats with::

	make latexdocs
	make epubdocs

Becoming A Kernel Developer
---------------------------

If you do not know anything about Linux kernel development, you should
look at the Linux KernelNewbies project:

	https://kernelnewbies.org

It consists of a helpful mailing list where you can ask almost any type
of basic kernel development question (make sure to search the archives
first, before asking something that has already been answered in the
past.)  It also has an IRC channel that you can use to ask questions in
real-time, and a lot of helpful documentation that is useful for
learning about Linux kernel development.

The website has basic information about code organization, subsystems,
and current projects (both in-tree and out-of-tree). It also describes
some basic logistical information, like how to compile a kernel and
apply a patch.

If you do not know where you want to start, but you want to look for
some task to start doing to join into the kernel development community,
go to the Linux Kernel Janitor's project:

	https://kernelnewbies.org/KernelJanitors

It is a great place to start.  It describes a list of relatively simple
problems that need to be cleaned up and fixed within the Linux kernel
source tree.  Working with the developers in charge of this project, you
will learn the basics of getting your patch into the Linux kernel tree,
and possibly be pointed in the direction of what to go work on next, if
you do not already have an idea.

Before making any actual modifications to the Linux kernel code, it is
imperative to understand how the code in question works.  For this
purpose, nothing is better than reading through it directly (most tricky
bits are commented well), perhaps even with the help of specialized
tools.  One such tool that is particularly recommended is the Linux
Cross-Reference project, which is able to present source code in a
self-referential, indexed webpage format. An excellent up-to-date
repository of the kernel code may be found at:

	https://elixir.bootlin.com/

The development process
-----------------------

Linux kernel development process currently consists of a few different
main kernel "branches" and lots of different subsystem-specific kernel
branches.  These different branches are:

  - Linus's mainline tree
  - Various stable trees with multiple major numbers
  - Subsystem-specific trees
  - linux-next integration testing tree

Mainline tree
~~~~~~~~~~~~~

The mainline tree is maintained by Linus Torvalds, and can be found at
https://kernel.org or in the repo.  Its development process is as follows:

  - As soon as a new kernel is released a two week window is open,
    during this period of time maintainers can submit big diffs to
    Linus, usually the patches that have already been included in the
    linux-next for a few weeks.  The preferred way to submit big changes
    is using git (the kernel's source management tool, more information
    can be found at https://git-scm.com/) but plain patches are also just
    fine.
  - After two weeks a -rc1 kernel is released and the focus is on making the
    new kernel as rock solid as possible.  Most of the patches at this point
    should fix a regression.  Bugs that have always existed are not
    regressions, so only push these kinds of fixes if they are important.
    Please note that a whole new driver (or filesystem) might be accepted
    after -rc1 because there is no risk of causing regressions with such a
    change as long as the change is self-contained and does not affect areas
    outside of the code that is being added.  git can be used to send
    patches to Linus after -rc1 is released, but the patches need to also be
    sent to a public mailing list for review.
  - A new -rc is released whenever Linus deems the current git tree to
    be in a reasonably sane state adequate for testing.  The goal is to
    release a new -rc kernel every week.
  - Process continues until the kernel is considered "ready", the
    process should last around 6 weeks.

It is worth mentioning what Andrew Morton wrote on the linux-kernel
mailing list about kernel releases:

	*"Nobody knows when a kernel will be released, because it's
	released according to perceived bug status, not according to a
	preconceived timeline."*

Various stable trees with multiple major numbers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Kernels with 3-part versions are -stable kernels. They contain
relatively small and critical fixes for security problems or significant
regressions discovered in a given major mainline release. Each release
in a major stable series increments the third part of the version
number, keeping the first two parts the same.

This is the recommended branch for users who want the most recent stable
kernel and are not interested in helping test development/experimental
versions.

Stable trees are maintained by the "stable" team <stable@vger.kernel.org>, and
are released as needs dictate.  The normal release period is approximately
two weeks, but it can be longer if there are no pressing problems.  A
security-related problem, instead, can cause a release to happen almost
instantly.

The file :ref:`Documentation/process/stable-kernel-rules.rst <stable_kernel_rules>`
in the kernel tree documents what kinds of changes are acceptable for
the -stable tree, and how the release process works.

Subsystem-specific trees
~~~~~~~~~~~~~~~~~~~~~~~~

The maintainers of the various kernel subsystems --- and also many
kernel subsystem developers --- expose their current state of
development in source repositories.  That way, others can see what is
happening in the different areas of the kernel.  In areas where
development is rapid, a developer may be asked to base his submissions
onto such a subsystem kernel tree so that conflicts between the
submission and other already ongoing work are avoided.

Most of these repositories are git trees, but there are also other SCMs
in use, or patch queues being published as quilt series.  Addresses of
these subsystem repositories are listed in the MAINTAINERS file.  Many
of them can be browsed at https://git.kernel.org/.

Before a proposed patch is committed to such a subsystem tree, it is
subject to review which primarily happens on mailing lists (see the
respective section below).  For several kernel subsystems, this review
process is tracked with the tool patchwork.  Patchwork offers a web
interface which shows patch postings, any comments on a patch or
revisions to it, and maintainers can mark patches as under review,
accepted, or rejected.  Most of these patchwork sites are listed at
https://patchwork.kernel.org/.

linux-next integration testing tree
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Before updates from subsystem trees are merged into the mainline tree,
they need to be integration-tested.  For this purpose, a special
testing repository exists into which virtually all subsystem trees are
pulled on an almost daily basis:

	https://git.kernel.org/?p=linux/kernel/git/next/linux-next.git

This way, the linux-next gives a summary outlook onto what will be
expected to go into the mainline kernel at the next merge period.
Adventurous testers are very welcome to runtime-test the linux-next.

Bug Reporting
-------------

The file 'Documentation/admin-guide/reporting-issues.rst' in the main kernel
source directory describes how to report a possible kernel bug, and details
what kind of information is needed by the kernel developers to help track
down the problem.

Managing bug reports
--------------------

One of the best ways to put into practice your hacking skills is by fixing
bugs reported by other people. Not only you will help to make the kernel
more stable, but you'll also learn to fix real world problems and you will
improve your skills, and other developers will be aware of your presence.
Fixing bugs is one of the best ways to get merits among other developers,
because not many people like wasting time fixing other people's bugs.

To work on already reported bug reports, find a subsystem you are interested in.
Check the MAINTAINERS file where bugs for that subsystem get reported to; often
it will be a mailing list, rarely a bugtracker. Search the archives of said
place for recent reports and help where you see fit. You may also want to check
https://bugzilla.kernel.org for bug reports; only a handful of kernel subsystems
use it actively for reporting or tracking, nevertheless bugs for the whole
kernel get filed there.

Mailing lists
-------------

As some of the above documents describe, the majority of the core kernel
developers participate on the Linux Kernel Mailing list.  Details on how
to subscribe and unsubscribe from the list can be found at:

	http://vger.kernel.org/vger-lists.html#linux-kernel

There are archives of the mailing list on the web in many different
places.  Use a search engine to find these archives.  For example:

	https://lore.kernel.org/lkml/

It is highly recommended that you search the archives about the topic
you want to bring up, before you post it to the list. A lot of things
already discussed in detail are only recorded at the mailing list
archives.

Most of the individual kernel subsystems also have their own separate
mailing list where they do their development efforts.  See the
MAINTAINERS file for a list of what these lists are for the different
groups.

Many of the lists are hosted on kernel.org. Information on them can be
found at:

	http://vger.kernel.org/vger-lists.html

Please remember to follow good behavioral habits when using the lists.
Though a bit cheesy, the following URL has some simple guidelines for
interacting with the list (or any list):

	http://www.albion.com/netiquette/

If multiple people respond to your mail, the CC: list of recipients may
get pretty large. Don't remove anybody from the CC: list without a good
reason, or don't reply only to the list address. Get used to receiving the
mail twice, one from the sender and the one from the list, and don't try
to tune that by adding fancy mail-headers, people will not like it.

Remember to keep the context and the attribution of your replies intact,
keep the "John Kernelhacker wrote ...:" lines at the top of your reply, and
add your statements between the individual quoted sections instead of
writing at the top of the mail.

If you add patches to your mail, make sure they are plain readable text
as stated in :ref:`Documentation/process/submitting-patches.rst <submittingpatches>`.
Kernel developers don't want to deal with
attachments or compressed patches; they may want to comment on
individual lines of your patch, which works only that way. Make sure you
use a mail program that does not mangle spaces and tab characters. A
good first test is to send the mail to yourself and try to apply your
own patch by yourself. If that doesn't work, get your mail program fixed
or change it until it works.

Above all, please remember to show respect to other subscribers.

Working with the community
--------------------------

The goal of the kernel community is to provide the best possible kernel
there is.  When you submit a patch for acceptance, it will be reviewed
on its technical merits and those alone.  So, what should you be
expecting?

  - criticism
  - comments
  - requests for change
  - requests for justification
  - silence

Remember, this is part of getting your patch into the kernel.  You have
to be able to take criticism and comments about your patches, evaluate
them at a technical level and either rework your patches or provide
clear and concise reasoning as to why those changes should not be made.
If there are no responses to your posting, wait a few days and try
again, sometimes things get lost in the huge volume.

What should you not do?

  - expect your patch to be accepted without question
  - become defensive
  - ignore comments
  - resubmit the patch without making any of the requested changes

In a community that is looking for the best technical solution possible,
there will always be differing opinions on how beneficial a patch is.
You have to be cooperative, and willing to adapt your idea to fit within
the kernel.  Or at least be willing to prove your idea is worth it.
Remember, being wrong is acceptable as long as you are willing to work
toward a solution that is right.

It is normal that the answers to your first patch might simply be a list
of a dozen things you should correct.  This does **not** imply that your
patch will not be accepted, and it is **not** meant against you
personally.  Simply correct all issues raised against your patch and
resend it.

Differences between the kernel community and corporate structures
-----------------------------------------------------------------

The kernel community works differently than most traditional corporate
development environments.  Here are a list of things that you can try to
do to avoid problems:

  Good things to say regarding your proposed changes:

    - "This solves multiple problems."
    - "This deletes 2000 lines of code."
    - "Here is a patch that explains what I am trying to describe."
    - "I tested it on 5 different architectures..."
    - "Here is a series of small patches that..."
    - "This increases performance on typical machines..."

  Bad things you should avoid saying:

    - "We did it this way in AIX/ptx/Solaris, so therefore it must be
      good..."
    - "I've being doing this for 20 years, so..."
    - "This is required for my company to make money"
    - "This is for our Enterprise product line."
    - "Here is my 1000 page design document that describes my idea"
    - "I've been working on this for 6 months..."
    - "Here's a 5000 line patch that..."
    - "I rewrote all of the current mess, and here it is..."
    - "I have a deadline, and this patch needs to be applied now."

Another way the kernel community is different than most traditional
software engineering work environments is the faceless nature of
interaction.  One benefit of using email and irc as the primary forms of
communication is the lack of discrimination based on gender or race.
The Linux kernel work environment is accepting of women and minorities
because all you are is an email address.  The international aspect also
helps to level the playing field because you can't guess gender based on
a person's name. A man may be named Andrea and a woman may be named Pat.
Most women who have worked in the Linux kernel and have expressed an
opinion have had positive experiences.

The language barrier can cause problems for some people who are not
comfortable with English.  A good grasp of the language can be needed in
order to get ideas across properly on mailing lists, so it is
recommended that you check your emails to make sure they make sense in
English before sending them.

Break up your changes
---------------------

The Linux kernel community does not gladly accept large chunks of code
dropped on it all at once.  The changes need to be properly introduced,
discussed, and broken up into tiny, individual portions.  This is almost
the exact opposite of what companies are used to doing.  Your proposal
should also be introduced very early in the development process, so that
you can receive feedback on what you are doing.  It also lets the
community feel that you are working with them, and not simply using them
as a dumping ground for your feature.  However, don't send 50 emails at
one time to a mailing list, your patch series should be smaller than
that almost all of the time.

The reasons for breaking things up are the following:

1) Small patches increase the likelihood that your patches will be
   applied, since they don't take much time or effort to verify for
   correctness.  A 5 line patch can be applied by a maintainer with
   barely a second glance. However, a 500 line patch may take hours to
   review for correctness (the time it takes is exponentially
   proportional to the size of the patch, or something).

   Small patches also make it very easy to debug when something goes
   wrong.  It's much easier to back out patches one by one than it is
   to dissect a very large patch after it's been applied (and broken
   something).

2) It's important not only to send small patches, but also to rewrite
   and simplify (or simply re-order) patches before submitting them.

Here is an analogy from kernel developer Al Viro:

	*"Think of a teacher grading homework from a math student.  The
	teacher does not want to see the student's trials and errors
	before they came up with the solution. They want to see the
	cleanest, most elegant answer.  A good student knows this, and
	would never submit her intermediate work before the final
	solution.*

	*The same is true of kernel development. The maintainers and
	reviewers do not want to see the thought process behind the
	solution to the problem one is solving. They want to see a
	simple and elegant solution."*

It may be challenging to keep the balance between presenting an elegant
solution and working together with the community and discussing your
unfinished work. Therefore it is good to get early in the process to
get feedback to improve your work, but also keep your changes in small
chunks that they may get already accepted, even when your whole task is
not ready for inclusion now.

Also realize that it is not acceptable to send patches for inclusion
that are unfinished and will be "fixed up later."

Justify your change
-------------------

Along with breaking up your patches, it is very important for you to let
the Linux community know why they should add this change.  New features
must be justified as being needed and useful.

Document your change
--------------------

When sending in your patches, pay special attention to what you say in
the text in your email.  This information will become the ChangeLog
information for the patch, and will be preserved for everyone to see for
all time.  It should describe the patch completely, containing:

  - why the change is necessary
  - the overall design approach in the patch
  - implementation details
  - testing results

For more details on what this should all look like, please see the
ChangeLog section of the document:

  "The Perfect Patch"
      https://www.ozlabs.org/~akpm/stuff/tpp.txt

All of these things are sometimes very hard to do. It can take years to
perfect these practices (if at all). It's a continuous process of
improvement that requires a lot of patience and determination. But
don't give up, it's possible. Many have done it before, and each had to
start exactly where you are now.

----------

Thanks to Paolo Ciarrocchi who allowed the "Development Process"
(https://lwn.net/Articles/94386/) section
to be based on text he had written, and to Randy Dunlap and Gerrit
Huizenga for some of the list of things you should and should not say.
Also thanks to Pat Mochel, Hanna Linder, Randy Dunlap, Kay Sievers,
Vojtech Pavlik, Jan Kara, Josh Boyer, Kees Cook, Andrew Morton, Andi
Kleen, Vadim Lobanov, Jesper Juhl, Adrian Bunk, Keri Harris, Frans Pop,
David A. Wheeler, Junio Hamano, Michael Kerrisk, and Alex Shepard for
their review, comments, and contributions.  Without their help, this
document would not have been possible.

Maintainer: Greg Kroah-Hartman <greg@kroah.com>

repo_name: angery/linux


code:
.. raw:: latex

	\renewcommand\thesection*
	\renewcommand\thesubsection*

.. _process_index:

=============================================
Working with the kernel development community
=============================================

So you want to be a Linux kernel developer?  Welcome!  While there is a lot
to be learned about the kernel in a technical sense, it is also important
to learn about how our community works.  Reading these documents will make
it much easier for you to get your changes merged with a minimum of
trouble.

Below are the essential guides that every developer should read.

.. toctree::
   :maxdepth: 1

   license-rules
   howto
   code-of-conduct
   code-of-conduct-interpretation
   development-process
   submitting-patches
   handling-regressions
   programming-language
   coding-style
   maintainer-handbooks
   maintainer-pgp-guide
   email-clients
   kernel-enforcement-statement
   kernel-driver-statement

Other guides to the community that are of interest to most developers are:

.. toctree::
   :maxdepth: 1

   changes
   stable-api-nonsense
   management-style
   stable-kernel-rules
   submit-checklist
   kernel-docs
   deprecated
   embargoed-hardware-issues
   maintainers
   researcher-guidelines

These are some overall technical guides that have been put here for now for
lack of a better place.

.. toctree::
   :maxdepth: 1

   applying-patches
   adding-syscalls
   magic-number
   volatile-considered-harmful
   botching-up-ioctls
   clang-format
   ../riscv/patch-acceptance
   ../core-api/unaligned-memory-access

.. only::  subproject and html

   Indices
   =======

   * :ref:`genindex`


code:
.. _kernel_docs:

Index of Further Kernel Documentation
=====================================

The need for a document like this one became apparent in the
linux-kernel mailing list as the same questions, asking for pointers
to information, appeared again and again.

Fortunately, as more and more people get to GNU/Linux, more and more
get interested in the Kernel. But reading the sources is not always
enough. It is easy to understand the code, but miss the concepts, the
philosophy and design decisions behind this code.

Unfortunately, not many documents are available for beginners to
start. And, even if they exist, there was no "well-known" place which
kept track of them. These lines try to cover this lack.

PLEASE, if you know any paper not listed here or write a new document,
include a reference to it here, following the kernel's patch submission
process. Any corrections, ideas or comments are also welcome.

All documents are cataloged with the following fields: the document's
"Title", the "Author"/s, the "URL" where they can be found, some
"Keywords" helpful when searching for specific topics, and a brief
"Description" of the Document.

.. note::

   The documents on each section of this document are ordered by its
   published date, from the newest to the oldest. The maintainer(s) should
   periodically retire resources as they become obsolte or outdated; with
   the exception of foundational books.

Docs at the Linux Kernel tree
-----------------------------

The Sphinx books should be built with ``make {htmldocs | pdfdocs | epubdocs}``.

    * Name: **linux/Documentation**

      :Author: Many.
      :Location: Documentation/
      :Keywords: text files, Sphinx.
      :Description: Documentation that comes with the kernel sources,
        inside the Documentation directory. Some pages from this document
        (including this document itself) have been moved there, and might
        be more up to date than the web version.

On-line docs
------------

    * Title: **Linux Kernel Mailing List Glossary**

      :Author: various
      :URL: https://kernelnewbies.org/KernelGlossary
      :Date: rolling version
      :Keywords: glossary, terms, linux-kernel.
      :Description: From the introduction: "This glossary is intended as
        a brief description of some of the acronyms and terms you may hear
        during discussion of the Linux kernel".

    * Title: **The Linux Kernel Module Programming Guide**

      :Author: Peter Jay Salzman, Michael Burian, Ori Pomerantz, Bob Mottram,
        Jim Huang.
      :URL: https://sysprog21.github.io/lkmpg/
      :Date: 2021
      :Keywords: modules, GPL book, /proc, ioctls, system calls,
        interrupt handlers .
      :Description: A very nice GPL book on the topic of modules
        programming. Lots of examples. Currently the new version is being
        actively maintained at https://github.com/sysprog21/lkmpg.

Published books
---------------

    * Title: **Linux Kernel Programming: A Comprehensive Guide to Kernel Internals, Writing Kernel Modules, and Kernel Synchronization**

          :Author: Kaiwan N. Billimoria
          :Publisher: Packt Publishing Ltd
          :Date: 2021
          :Pages: 754
          :ISBN: 978-1789953435

    * Title: **Linux Kernel Development, 3rd Edition**

      :Author: Robert Love
      :Publisher: Addison-Wesley
      :Date: July, 2010
      :Pages: 440
      :ISBN: 978-0672329463
      :Notes: Foundational book

.. _ldd3_published:

    * Title: **Linux Device Drivers, 3rd Edition**

      :Authors: Jonathan Corbet, Alessandro Rubini, and Greg Kroah-Hartman
      :Publisher: O'Reilly & Associates
      :Date: 2005
      :Pages: 636
      :ISBN: 0-596-00590-3
      :Notes: Foundational book. Further information in
        http://www.oreilly.com/catalog/linuxdrive3/
        PDF format, URL: https://lwn.net/Kernel/LDD3/

    * Title: **The Design of the UNIX Operating System**

      :Author: Maurice J. Bach
      :Publisher: Prentice Hall
      :Date: 1986
      :Pages: 471
      :ISBN: 0-13-201757-1
      :Notes: Foundational book

Miscellaneous
-------------

    * Name: **Cross-Referencing Linux**

      :URL: https://elixir.bootlin.com/
      :Keywords: Browsing source code.
      :Description: Another web-based Linux kernel source code browser.
        Lots of cross references to variables and functions. You can see
        where they are defined and where they are used.

    * Name: **Linux Weekly News**

      :URL: https://lwn.net
      :Keywords: latest kernel news.
      :Description: The title says it all. There's a fixed kernel section
        summarizing developers' work, bug fixes, new features and versions
        produced during the week.

    * Name: **The home page of Linux-MM**

      :Author: The Linux-MM team.
      :URL: https://linux-mm.org/
      :Keywords: memory management, Linux-MM, mm patches, TODO, docs,
        mailing list.
      :Description: Site devoted to Linux Memory Management development.
        Memory related patches, HOWTOs, links, mm developers... Don't miss
        it if you are interested in memory management development!

    * Name: **Kernel Newbies IRC Channel and Website**

      :URL: https://www.kernelnewbies.org
      :Keywords: IRC, newbies, channel, asking doubts.
      :Description: #kernelnewbies on irc.oftc.net.
        #kernelnewbies is an IRC network dedicated to the 'newbie'
        kernel hacker. The audience mostly consists of people who are
        learning about the kernel, working on kernel projects or
        professional kernel hackers that want to help less seasoned kernel
        people.
        #kernelnewbies is on the OFTC IRC Network.
        Try irc.oftc.net as your server and then /join #kernelnewbies.
        The kernelnewbies website also hosts articles, documents, FAQs...

    * Name: **linux-kernel mailing list archives and search engines**

      :URL: http://vger.kernel.org/vger-lists.html
      :URL: http://www.uwsg.indiana.edu/hypermail/linux/kernel/index.html
      :URL: http://groups.google.com/group/mlist.linux.kernel
      :Keywords: linux-kernel, archives, search.
      :Description: Some of the linux-kernel mailing list archivers. If
        you have a better/another one, please let me know.

-------

This document was originally based on:

 https://www.dit.upm.es/~jmseyas/linux/kernel/hackers-docs.html

and written by Juan-Mariano de Goyeneche


code:
.. _process_statement_driver:

Kernel Driver Statement
-----------------------

Position Statement on Linux Kernel Modules
==========================================

We, the undersigned Linux kernel developers, consider any closed-source
Linux kernel module or driver to be harmful and undesirable. We have
repeatedly found them to be detrimental to Linux users, businesses, and
the greater Linux ecosystem. Such modules negate the openness,
stability, flexibility, and maintainability of the Linux development
model and shut their users off from the expertise of the Linux
community. Vendors that provide closed-source kernel modules force their
customers to give up key Linux advantages or choose new vendors.
Therefore, in order to take full advantage of the cost savings and
shared support benefits open source has to offer, we urge vendors to
adopt a policy of supporting their customers on Linux with open-source
kernel code.

We speak only for ourselves, and not for any company we might work for
today, have in the past, or will in the future.

 - Dave Airlie
 - Nick Andrew
 - Jens Axboe
 - Ralf Baechle
 - Felipe Balbi
 - Ohad Ben-Cohen
 - Muli Ben-Yehuda
 - Jiri Benc
 - Arnd Bergmann
 - Thomas Bogendoerfer
 - Vitaly Bordug
 - James Bottomley
 - Josh Boyer
 - Neil Brown
 - Mark Brown
 - David Brownell
 - Michael Buesch
 - Franck Bui-Huu
 - Adrian Bunk
 - François Cami
 - Ralph Campbell
 - Luiz Fernando N. Capitulino
 - Mauro Carvalho Chehab
 - Denis Cheng
 - Jonathan Corbet
 - Glauber Costa
 - Alan Cox
 - Magnus Damm
 - Ahmed S. Darwish
 - Robert P. J. Day
 - Hans de Goede
 - Arnaldo Carvalho de Melo
 - Helge Deller
 - Jean Delvare
 - Mathieu Desnoyers
 - Sven-Thorsten Dietrich
 - Alexey Dobriyan
 - Daniel Drake
 - Alex Dubov
 - Randy Dunlap
 - Michael Ellerman
 - Pekka Enberg
 - Jan Engelhardt
 - Mark Fasheh
 - J. Bruce Fields
 - Larry Finger
 - Jeremy Fitzhardinge
 - Mike Frysinger
 - Kumar Gala
 - Robin Getz
 - Liam Girdwood
 - Jan-Benedict Glaw
 - Thomas Gleixner
 - Brice Goglin
 - Cyrill Gorcunov
 - Andy Gospodarek
 - Thomas Graf
 - Krzysztof Halasa
 - Harvey Harrison
 - Stephen Hemminger
 - Michael Hennerich
 - Tejun Heo
 - Benjamin Herrenschmidt
 - Kristian Høgsberg
 - Henrique de Moraes Holschuh
 - Marcel Holtmann
 - Mike Isely
 - Takashi Iwai
 - Olof Johansson
 - Dave Jones
 - Jesper Juhl
 - Matthias Kaehlcke
 - Kenji Kaneshige
 - Jan Kara
 - Jeremy Kerr
 - Russell King
 - Olaf Kirch
 - Roel Kluin
 - Hans-Jürgen Koch
 - Auke Kok
 - Peter Korsgaard
 - Jiri Kosina
 - Aaro Koskinen
 - Mariusz Kozlowski
 - Greg Kroah-Hartman
 - Michael Krufky
 - Aneesh Kumar
 - Clemens Ladisch
 - Christoph Lameter
 - Gunnar Larisch
 - Anders Larsen
 - Grant Likely
 - John W. Linville
 - Yinghai Lu
 - Tony Luck
 - Pavel Machek
 - Matt Mackall
 - Paul Mackerras
 - Roland McGrath
 - Patrick McHardy
 - Kyle McMartin
 - Paul Menage
 - Thierry Merle
 - Eric Miao
 - Akinobu Mita
 - Ingo Molnar
 - James Morris
 - Andrew Morton
 - Paul Mundt
 - Oleg Nesterov
 - Luca Olivetti
 - S.Çağlar Onur
 - Pierre Ossman
 - Keith Owens
 - Venkatesh Pallipadi
 - Nick Piggin
 - Nicolas Pitre
 - Evgeniy Polyakov
 - Richard Purdie
 - Mike Rapoport
 - Sam Ravnborg
 - Gerrit Renker
 - Stefan Richter
 - David Rientjes
 - Luis R. Rodriguez
 - Stefan Roese
 - Francois Romieu
 - Rami Rosen
 - Stephen Rothwell
 - Maciej W. Rozycki
 - Mark Salyzyn
 - Yoshinori Sato
 - Deepak Saxena
 - Holger Schurig
 - Amit Shah
 - Yoshihiro Shimoda
 - Sergei Shtylyov
 - Kay Sievers
 - Sebastian Siewior
 - Rik Snel
 - Jes Sorensen
 - Alexey Starikovskiy
 - Alan Stern
 - Timur Tabi
 - Hirokazu Takata
 - Eliezer Tamir
 - Eugene Teo
 - Doug Thompson
 - FUJITA Tomonori
 - Dmitry Torokhov
 - Marcelo Tosatti
 - Steven Toth
 - Theodore Tso
 - Matthias Urlichs
 - Geert Uytterhoeven
 - Arjan van de Ven
 - Ivo van Doorn
 - Rik van Riel
 - Wim Van Sebroeck
 - Hans Verkuil
 - Horst H. von Brand
 - Dmitri Vorobiev
 - Anton Vorontsov
 - Daniel Walker
 - Johannes Weiner
 - Harald Welte
 - Matthew Wilcox
 - Dan J. Williams
 - Darrick J. Wong
 - David Woodhouse
 - Chris Wright
 - Bryan Wu
 - Rafael J. Wysocki
 - Herbert Xu
 - Vlad Yasevich
 - Peter Zijlstra
 - Bartlomiej Zolnierkiewicz


code:
.. _process_statement_kernel:

Linux Kernel Enforcement Statement
----------------------------------

As developers of the Linux kernel, we have a keen interest in how our software
is used and how the license for our software is enforced.  Compliance with the
reciprocal sharing obligations of GPL-2.0 is critical to the long-term
sustainability of our software and community.

Although there is a right to enforce the separate copyright interests in the
contributions made to our community, we share an interest in ensuring that
individual enforcement actions are conducted in a manner that benefits our
community and do not have an unintended negative impact on the health and
growth of our software ecosystem.  In order to deter unhelpful enforcement
actions, we agree that it is in the best interests of our development
community to undertake the following commitment to users of the Linux kernel
on behalf of ourselves and any successors to our copyright interests:

    Notwithstanding the termination provisions of the GPL-2.0, we agree that
    it is in the best interests of our development community to adopt the
    following provisions of GPL-3.0 as additional permissions under our
    license with respect to any non-defensive assertion of rights under the
    license.

	However, if you cease all violation of this License, then your license
	from a particular copyright holder is reinstated (a) provisionally,
	unless and until the copyright holder explicitly and finally
	terminates your license, and (b) permanently, if the copyright holder
	fails to notify you of the violation by some reasonable means prior to
	60 days after the cessation.

	Moreover, your license from a particular copyright holder is
	reinstated permanently if the copyright holder notifies you of the
	violation by some reasonable means, this is the first time you have
	received notice of violation of this License (for any work) from that
	copyright holder, and you cure the violation prior to 30 days after
	your receipt of the notice.

Our intent in providing these assurances is to encourage more use of the
software.  We want companies and individuals to use, modify and distribute
this software.  We want to work with users in an open and transparent way to
eliminate any uncertainty about our expectations regarding compliance or
enforcement that might limit adoption of our software.  We view legal action
as a last resort, to be initiated only when other community efforts have
failed to resolve the problem.

Finally, once a non-compliance issue is resolved, we hope the user will feel
welcome to join us in our efforts on this project.  Working together, we will
be stronger.

Except where noted below, we speak only for ourselves, and not for any company
we might work for today, have in the past, or will in the future.

  - Laura Abbott
  - Bjorn Andersson (Linaro)
  - Andrea Arcangeli
  - Neil Armstrong
  - Jens Axboe
  - Pablo Neira Ayuso
  - Khalid Aziz
  - Ralf Baechle
  - Felipe Balbi
  - Arnd Bergmann
  - Ard Biesheuvel
  - Tim Bird
  - Paolo Bonzini
  - Christian Borntraeger
  - Mark Brown (Linaro)
  - Paul Burton
  - Javier Martinez Canillas
  - Rob Clark
  - Kees Cook (Google)
  - Jonathan Corbet
  - Dennis Dalessandro
  - Vivien Didelot (Savoir-faire Linux)
  - Hans de Goede
  - Mel Gorman (SUSE)
  - Sven Eckelmann
  - Alex Elder (Linaro)
  - Fabio Estevam
  - Larry Finger
  - Bhumika Goyal
  - Andy Gross
  - Juergen Gross
  - Shawn Guo
  - Ulf Hansson
  - Stephen Hemminger (Microsoft)
  - Tejun Heo
  - Rob Herring
  - Masami Hiramatsu
  - Michal Hocko
  - Simon Horman
  - Johan Hovold (Hovold Consulting AB)
  - Christophe JAILLET
  - Olof Johansson
  - Lee Jones (Linaro)
  - Heiner Kallweit
  - Srinivas Kandagatla
  - Jan Kara
  - Shuah Khan (Samsung)
  - David Kershner
  - Jaegeuk Kim
  - Namhyung Kim
  - Colin Ian King
  - Jeff Kirsher
  - Greg Kroah-Hartman (Linux Foundation)
  - Christian König
  - Vinod Koul
  - Krzysztof Kozlowski
  - Viresh Kumar
  - Aneesh Kumar K.V
  - Julia Lawall
  - Doug Ledford
  - Chuck Lever (Oracle)
  - Daniel Lezcano
  - Shaohua Li
  - Xin Long
  - Tony Luck
  - Catalin Marinas (Arm Ltd)
  - Mike Marshall
  - Chris Mason
  - Paul E. McKenney
  - Arnaldo Carvalho de Melo
  - David S. Miller
  - Ingo Molnar
  - Kuninori Morimoto
  - Trond Myklebust
  - Martin K. Petersen (Oracle)
  - Borislav Petkov
  - Jiri Pirko
  - Josh Poimboeuf
  - Sebastian Reichel (Collabora)
  - Guenter Roeck
  - Joerg Roedel
  - Leon Romanovsky
  - Steven Rostedt (VMware)
  - Frank Rowand
  - Ivan Safonov
  - Anna Schumaker
  - Jes Sorensen
  - K.Y. Srinivasan
  - David Sterba (SUSE)
  - Heiko Stuebner
  - Jiri Kosina (SUSE)
  - Willy Tarreau
  - Dmitry Torokhov
  - Linus Torvalds
  - Thierry Reding
  - Rik van Riel
  - Luis R. Rodriguez
  - Geert Uytterhoeven (Glider bvba)
  - Eduardo Valentin (Amazon.com)
  - Daniel Vetter
  - Linus Walleij
  - Richard Weinberger
  - Dan Williams
  - Rafael J. Wysocki
  - Arvind Yadav
  - Masahiro Yamada
  - Wei Yongjun
  - Lv Zheng
  - Marc Zyngier (Arm Ltd)


code:
.. SPDX-License-Identifier: GPL-2.0

.. _kernel_licensing:

Linux kernel licensing rules
============================

The Linux Kernel is provided under the terms of the GNU General Public
License version 2 only (GPL-2.0), as provided in LICENSES/preferred/GPL-2.0,
with an explicit syscall exception described in
LICENSES/exceptions/Linux-syscall-note, as described in the COPYING file.

This documentation file provides a description of how each source file
should be annotated to make its license clear and unambiguous.
It doesn't replace the Kernel's license.

The license described in the COPYING file applies to the kernel source
as a whole, though individual source files can have a different license
which is required to be compatible with the GPL-2.0::

    GPL-1.0+  :  GNU General Public License v1.0 or later
    GPL-2.0+  :  GNU General Public License v2.0 or later
    LGPL-2.0  :  GNU Library General Public License v2 only
    LGPL-2.0+ :  GNU Library General Public License v2 or later
    LGPL-2.1  :  GNU Lesser General Public License v2.1 only
    LGPL-2.1+ :  GNU Lesser General Public License v2.1 or later

Aside from that, individual files can be provided under a dual license,
e.g. one of the compatible GPL variants and alternatively under a
permissive license like BSD, MIT etc.

The User-space API (UAPI) header files, which describe the interface of
user-space programs to the kernel are a special case.  According to the
note in the kernel COPYING file, the syscall interface is a clear boundary,
which does not extend the GPL requirements to any software which uses it to
communicate with the kernel.  Because the UAPI headers must be includable
into any source files which create an executable running on the Linux
kernel, the exception must be documented by a special license expression.

The common way of expressing the license of a source file is to add the
matching boilerplate text into the top comment of the file.  Due to
formatting, typos etc. these "boilerplates" are hard to validate for
tools which are used in the context of license compliance.

An alternative to boilerplate text is the use of Software Package Data
Exchange (SPDX) license identifiers in each source file.  SPDX license
identifiers are machine parsable and precise shorthands for the license
under which the content of the file is contributed.  SPDX license
identifiers are managed by the SPDX Workgroup at the Linux Foundation and
have been agreed on by partners throughout the industry, tool vendors, and
legal teams.  For further information see https://spdx.org/

The Linux kernel requires the precise SPDX identifier in all source files.
The valid identifiers used in the kernel are explained in the section
`License identifiers`_ and have been retrieved from the official SPDX
license list at https://spdx.org/licenses/ along with the license texts.

License identifier syntax
-------------------------

1. Placement:

   The SPDX license identifier in kernel files shall be added at the first
   possible line in a file which can contain a comment.  For the majority
   of files this is the first line, except for scripts which require the
   '#!PATH_TO_INTERPRETER' in the first line.  For those scripts the SPDX
   identifier goes into the second line.

|

2. Style:

   The SPDX license identifier is added in form of a comment.  The comment
   style depends on the file type::

      C source:	// SPDX-License-Identifier: <SPDX License Expression>
      C header:	/* SPDX-License-Identifier: <SPDX License Expression> */
      ASM:	/* SPDX-License-Identifier: <SPDX License Expression> */
      scripts:	# SPDX-License-Identifier: <SPDX License Expression>
      .rst:	.. SPDX-License-Identifier: <SPDX License Expression>
      .dts{i}:	// SPDX-License-Identifier: <SPDX License Expression>

   If a specific tool cannot handle the standard comment style, then the
   appropriate comment mechanism which the tool accepts shall be used. This
   is the reason for having the "/\* \*/" style comment in C header
   files. There was build breakage observed with generated .lds files where
   'ld' failed to parse the C++ comment. This has been fixed by now, but
   there are still older assembler tools which cannot handle C++ style
   comments.

|

3. Syntax:

   A <SPDX License Expression> is either an SPDX short form license
   identifier found on the SPDX License List, or the combination of two
   SPDX short form license identifiers separated by "WITH" when a license
   exception applies. When multiple licenses apply, an expression consists
   of keywords "AND", "OR" separating sub-expressions and surrounded by
   "(", ")" .

   License identifiers for licenses like [L]GPL with the 'or later' option
   are constructed by using a "+" for indicating the 'or later' option.::

      // SPDX-License-Identifier: GPL-2.0+
      // SPDX-License-Identifier: LGPL-2.1+

   WITH should be used when there is a modifier to a license needed.
   For example, the linux kernel UAPI files use the expression::

      // SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note
      // SPDX-License-Identifier: GPL-2.0+ WITH Linux-syscall-note

   Other examples using WITH exceptions found in the kernel are::

      // SPDX-License-Identifier: GPL-2.0 WITH mif-exception
      // SPDX-License-Identifier: GPL-2.0+ WITH GCC-exception-2.0

   Exceptions can only be used with particular License identifiers. The
   valid License identifiers are listed in the tags of the exception text
   file. For details see the point `Exceptions`_ in the chapter `License
   identifiers`_.

   OR should be used if the file is dual licensed and only one license is
   to be selected.  For example, some dtsi files are available under dual
   licenses::

      // SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause

   Examples from the kernel for license expressions in dual licensed files::

      // SPDX-License-Identifier: GPL-2.0 OR MIT
      // SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause
      // SPDX-License-Identifier: GPL-2.0 OR Apache-2.0
      // SPDX-License-Identifier: GPL-2.0 OR MPL-1.1
      // SPDX-License-Identifier: (GPL-2.0 WITH Linux-syscall-note) OR MIT
      // SPDX-License-Identifier: GPL-1.0+ OR BSD-3-Clause OR OpenSSL

   AND should be used if the file has multiple licenses whose terms all
   apply to use the file. For example, if code is inherited from another
   project and permission has been given to put it in the kernel, but the
   original license terms need to remain in effect::

      // SPDX-License-Identifier: (GPL-2.0 WITH Linux-syscall-note) AND MIT

   Another other example where both sets of license terms need to be
   adhered to is::

      // SPDX-License-Identifier: GPL-1.0+ AND LGPL-2.1+

License identifiers
-------------------

The licenses currently used, as well as the licenses for code added to the
kernel, can be broken down into:

1. _`Preferred licenses`:

   Whenever possible these licenses should be used as they are known to be
   fully compatible and widely used.  These licenses are available from the
   directory::

      LICENSES/preferred/

   in the kernel source tree.

   The files in this directory contain the full license text and
   `Metatags`_.  The file names are identical to the SPDX license
   identifier which shall be used for the license in source files.

   Examples::

      LICENSES/preferred/GPL-2.0

   Contains the GPL version 2 license text and the required metatags::

      LICENSES/preferred/MIT

   Contains the MIT license text and the required metatags

   _`Metatags`:

   The following meta tags must be available in a license file:

   - Valid-License-Identifier:

     One or more lines which declare which License Identifiers are valid
     inside the project to reference this particular license text.  Usually
     this is a single valid identifier, but e.g. for licenses with the 'or
     later' options two identifiers are valid.

   - SPDX-URL:

     The URL of the SPDX page which contains additional information related
     to the license.

   - Usage-Guidance:

     Freeform text for usage advice. The text must include correct examples
     for the SPDX license identifiers as they should be put into source
     files according to the `License identifier syntax`_ guidelines.

   - License-Text:

     All text after this tag is treated as the original license text

   File format examples::

      Valid-License-Identifier: GPL-2.0
      Valid-License-Identifier: GPL-2.0+
      SPDX-URL: https://spdx.org/licenses/GPL-2.0.html
      Usage-Guide:
        To use this license in source code, put one of the following SPDX
	tag/value pairs into a comment according to the placement
	guidelines in the licensing rules documentation.
	For 'GNU General Public License (GPL) version 2 only' use:
	  SPDX-License-Identifier: GPL-2.0
	For 'GNU General Public License (GPL) version 2 or any later version' use:
	  SPDX-License-Identifier: GPL-2.0+
      License-Text:
        Full license text

   ::

      SPDX-License-Identifier: MIT
      SPDX-URL: https://spdx.org/licenses/MIT.html
      Usage-Guide:
	To use this license in source code, put the following SPDX
	tag/value pair into a comment according to the placement
	guidelines in the licensing rules documentation.
	  SPDX-License-Identifier: MIT
      License-Text:
        Full license text

|

2. Deprecated licenses:

   These licenses should only be used for existing code or for importing
   code from a different project.  These licenses are available from the
   directory::

      LICENSES/deprecated/

   in the kernel source tree.

   The files in this directory contain the full license text and
   `Metatags`_.  The file names are identical to the SPDX license
   identifier which shall be used for the license in source files.

   Examples::

      LICENSES/deprecated/ISC

   Contains the Internet Systems Consortium license text and the required
   metatags::

      LICENSES/deprecated/GPL-1.0

   Contains the GPL version 1 license text and the required metatags.

   Metatags:

   The metatag requirements for 'other' licenses are identical to the
   requirements of the `Preferred licenses`_.

   File format example::

      Valid-License-Identifier: ISC
      SPDX-URL: https://spdx.org/licenses/ISC.html
      Usage-Guide:
        Usage of this license in the kernel for new code is discouraged
	and it should solely be used for importing code from an already
	existing project.
        To use this license in source code, put the following SPDX
	tag/value pair into a comment according to the placement
	guidelines in the licensing rules documentation.
	  SPDX-License-Identifier: ISC
      License-Text:
        Full license text

|

3. Dual Licensing Only

   These licenses should only be used to dual license code with another
   license in addition to a preferred license.  These licenses are available
   from the directory::

      LICENSES/dual/

   in the kernel source tree.

   The files in this directory contain the full license text and
   `Metatags`_.  The file names are identical to the SPDX license
   identifier which shall be used for the license in source files.

   Examples::

      LICENSES/dual/MPL-1.1

   Contains the Mozilla Public License version 1.1 license text and the
   required metatags::

      LICENSES/dual/Apache-2.0

   Contains the Apache License version 2.0 license text and the required
   metatags.

   Metatags:

   The metatag requirements for 'other' licenses are identical to the
   requirements of the `Preferred licenses`_.

   File format example::

      Valid-License-Identifier: MPL-1.1
      SPDX-URL: https://spdx.org/licenses/MPL-1.1.html
      Usage-Guide:
        Do NOT use. The MPL-1.1 is not GPL2 compatible. It may only be used for
        dual-licensed files where the other license is GPL2 compatible.
        If you end up using this it MUST be used together with a GPL2 compatible
        license using "OR".
        To use the Mozilla Public License version 1.1 put the following SPDX
        tag/value pair into a comment according to the placement guidelines in
        the licensing rules documentation:
      SPDX-License-Identifier: MPL-1.1
      License-Text:
        Full license text

|

4. _`Exceptions`:

   Some licenses can be amended with exceptions which grant certain rights
   which the original license does not.  These exceptions are available
   from the directory::

      LICENSES/exceptions/

   in the kernel source tree.  The files in this directory contain the full
   exception text and the required `Exception Metatags`_.

   Examples::

      LICENSES/exceptions/Linux-syscall-note

   Contains the Linux syscall exception as documented in the COPYING
   file of the Linux kernel, which is used for UAPI header files.
   e.g. /\* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note \*/::

      LICENSES/exceptions/GCC-exception-2.0

   Contains the GCC 'linking exception' which allows to link any binary
   independent of its license against the compiled version of a file marked
   with this exception. This is required for creating runnable executables
   from source code which is not compatible with the GPL.

   _`Exception Metatags`:

   The following meta tags must be available in an exception file:

   - SPDX-Exception-Identifier:

     One exception identifier which can be used with SPDX license
     identifiers.

   - SPDX-URL:

     The URL of the SPDX page which contains additional information related
     to the exception.

   - SPDX-Licenses:

     A comma separated list of SPDX license identifiers for which the
     exception can be used.

   - Usage-Guidance:

     Freeform text for usage advice. The text must be followed by correct
     examples for the SPDX license identifiers as they should be put into
     source files according to the `License identifier syntax`_ guidelines.

   - Exception-Text:

     All text after this tag is treated as the original exception text

   File format examples::

      SPDX-Exception-Identifier: Linux-syscall-note
      SPDX-URL: https://spdx.org/licenses/Linux-syscall-note.html
      SPDX-Licenses: GPL-2.0, GPL-2.0+, GPL-1.0+, LGPL-2.0, LGPL-2.0+, LGPL-2.1, LGPL-2.1+
      Usage-Guidance:
        This exception is used together with one of the above SPDX-Licenses
	to mark user-space API (uapi) header files so they can be included
	into non GPL compliant user-space application code.
        To use this exception add it with the keyword WITH to one of the
	identifiers in the SPDX-Licenses tag:
	  SPDX-License-Identifier: <SPDX-License> WITH Linux-syscall-note
      Exception-Text:
        Full exception text

   ::

      SPDX-Exception-Identifier: GCC-exception-2.0
      SPDX-URL: https://spdx.org/licenses/GCC-exception-2.0.html
      SPDX-Licenses: GPL-2.0, GPL-2.0+
      Usage-Guidance:
        The "GCC Runtime Library exception 2.0" is used together with one
	of the above SPDX-Licenses for code imported from the GCC runtime
	library.
        To use this exception add it with the keyword WITH to one of the
	identifiers in the SPDX-Licenses tag:
	  SPDX-License-Identifier: <SPDX-License> WITH GCC-exception-2.0
      Exception-Text:
        Full exception text

All SPDX license identifiers and exceptions must have a corresponding file
in the LICENSES subdirectories. This is required to allow tool
verification (e.g. checkpatch.pl) and to have the licenses ready to read
and extract right from the source, which is recommended by various FOSS
organizations, e.g. the `FSFE REUSE initiative <https://reuse.software/>`_.

_`MODULE_LICENSE`
-----------------

   Loadable kernel modules also require a MODULE_LICENSE() tag. This tag is
   neither a replacement for proper source code license information
   (SPDX-License-Identifier) nor in any way relevant for expressing or
   determining the exact license under which the source code of the module
   is provided.

   The sole purpose of this tag is to provide sufficient information
   whether the module is free software or proprietary for the kernel
   module loader and for user space tools.

   The valid license strings for MODULE_LICENSE() are:

    ============================= =============================================
    "GPL"			  Module is licensed under GPL version 2. This
				  does not express any distinction between
				  GPL-2.0-only or GPL-2.0-or-later. The exact
				  license information can only be determined
				  via the license information in the
				  corresponding source files.

    "GPL v2"			  Same as "GPL". It exists for historic
				  reasons.

    "GPL and additional rights"   Historical variant of expressing that the
				  module source is dual licensed under a
				  GPL v2 variant and MIT license. Please do
				  not use in new code.

    "Dual MIT/GPL"		  The correct way of expressing that the
				  module is dual licensed under a GPL v2
				  variant or MIT license choice.

    "Dual BSD/GPL"		  The module is dual licensed under a GPL v2
				  variant or BSD license choice. The exact
				  variant of the BSD license can only be
				  determined via the license information
				  in the corresponding source files.

    "Dual MPL/GPL"		  The module is dual licensed under a GPL v2
				  variant or Mozilla Public License (MPL)
				  choice. The exact variant of the MPL
				  license can only be determined via the
				  license information in the corresponding
				  source files.

    "Proprietary"		  The module is under a proprietary license.
				  This string is solely for proprietary third
				  party modules and cannot be used for modules
				  which have their source code in the kernel
				  tree. Modules tagged that way are tainting
				  the kernel with the 'P' flag when loaded and
				  the kernel module loader refuses to link such
				  modules against symbols which are exported
				  with EXPORT_SYMBOL_GPL().
    ============================= =============================================