code:
=======================================
How to use dm-crypt and swsusp together
=======================================

Author: Andreas Steinmetz <ast@domdv.de>

Some prerequisites:
You know how dm-crypt works. If not, visit the following web page:
http://www.saout.de/misc/dm-crypt/
You have read Documentation/power/swsusp.rst and understand it.
You did read Documentation/admin-guide/initrd.rst and know how an initrd works.
You know how to create or how to modify an initrd.

Now your system is properly set up, your disk is encrypted except for
the swap device(s) and the boot partition which may contain a mini
system for crypto setup and/or rescue purposes. You may even have
an initrd that does your current crypto setup already.

At this point you want to encrypt your swap, too. Still you want to
be able to suspend using swsusp. This, however, means that you
have to be able to either enter a passphrase or that you read
the key(s) from an external device like a pcmcia flash disk
or an usb stick prior to resume. So you need an initrd, that sets
up dm-crypt and then asks swsusp to resume from the encrypted
swap device.

The most important thing is that you set up dm-crypt in such
a way that the swap device you suspend to/resume from has
always the same major/minor within the initrd as well as
within your running system. The easiest way to achieve this is
to always set up this swap device first with dmsetup, so that
it will always look like the following::

  brw-------  1 root root 254, 0 Jul 28 13:37 /dev/mapper/swap0

Now set up your kernel to use /dev/mapper/swap0 as the default
resume partition, so your kernel .config contains::

  CONFIG_PM_STD_PARTITION="/dev/mapper/swap0"

Prepare your boot loader to use the initrd you will create or
modify. For lilo the simplest setup looks like the following
lines::

  image=/boot/vmlinuz
  initrd=/boot/initrd.gz
  label=linux
  append="root=/dev/ram0 init=/linuxrc rw"

Finally you need to create or modify your initrd. Lets assume
you create an initrd that reads the required dm-crypt setup
from a pcmcia flash disk card. The card is formatted with an ext2
fs which resides on /dev/hde1 when the card is inserted. The
card contains at least the encrypted swap setup in a file
named "swapkey". /etc/fstab of your initrd contains something
like the following::

  /dev/hda1   /mnt    ext3      ro                            0 0
  none        /proc   proc      defaults,noatime,nodiratime   0 0
  none        /sys    sysfs     defaults,noatime,nodiratime   0 0

/dev/hda1 contains an unencrypted mini system that sets up all
of your crypto devices, again by reading the setup from the
pcmcia flash disk. What follows now is a /linuxrc for your
initrd that allows you to resume from encrypted swap and that
continues boot with your mini system on /dev/hda1 if resume
does not happen::

  #!/bin/sh
  PATH=/sbin:/bin:/usr/sbin:/usr/bin
  mount /proc
  mount /sys
  mapped=0
  noresume=`grep -c noresume /proc/cmdline`
  if [ "$*" != "" ]
  then
    noresume=1
  fi
  dmesg -n 1
  /sbin/cardmgr -q
  for i in 1 2 3 4 5 6 7 8 9 0
  do
    if [ -f /proc/ide/hde/media ]
    then
      usleep 500000
      mount -t ext2 -o ro /dev/hde1 /mnt
      if [ -f /mnt/swapkey ]
      then
        dmsetup create swap0 /mnt/swapkey > /dev/null 2>&1 && mapped=1
      fi
      umount /mnt
      break
    fi
    usleep 500000
  done
  killproc /sbin/cardmgr
  dmesg -n 6
  if [ $mapped = 1 ]
  then
    if [ $noresume != 0 ]
    then
      mkswap /dev/mapper/swap0 > /dev/null 2>&1
    fi
    echo 254:0 > /sys/power/resume
    dmsetup remove swap0
  fi
  umount /sys
  mount /mnt
  umount /proc
  cd /mnt
  pivot_root . mnt
  mount /proc
  umount -l /mnt
  umount /proc
  exec chroot . /sbin/init $* < dev/console > dev/console 2>&1

Please don't mind the weird loop above, busybox's msh doesn't know
the let statement. Now, what is happening in the script?
First we have to decide if we want to try to resume, or not.
We will not resume if booting with "noresume" or any parameters
for init like "single" or "emergency" as boot parameters.

Then we need to set up dmcrypt with the setup data from the
pcmcia flash disk. If this succeeds we need to reset the swap
device if we don't want to resume. The line "echo 254:0 > /sys/power/resume"
then attempts to resume from the first device mapper device.
Note that it is important to set the device in /sys/power/resume,
regardless if resuming or not, otherwise later suspend will fail.
If resume starts, script execution terminates here.

Otherwise we just remove the encrypted swap device and leave it to the
mini system on /dev/hda1 to set the whole crypto up (it is up to
you to modify this to your taste).

What then follows is the well known process to change the root
file system and continue booting from there. I prefer to unmount
the initrd prior to continue booting but it is up to you to modify
this.


code:
============
Swap suspend
============

Some warnings, first.

.. warning::

   **BIG FAT WARNING**

   If you touch anything on disk between suspend and resume...
				...kiss your data goodbye.

   If you do resume from initrd after your filesystems are mounted...
				...bye bye root partition.

			[this is actually same case as above]

   If you have unsupported ( ) devices using DMA, you may have some
   problems. If your disk driver does not support suspend... (IDE does),
   it may cause some problems, too. If you change kernel command line
   between suspend and resume, it may do something wrong. If you change
   your hardware while system is suspended... well, it was not good idea;
   but it will probably only crash.

   ( ) suspend/resume support is needed to make it safe.

   If you have any filesystems on USB devices mounted before software suspend,
   they won't be accessible after resume and you may lose data, as though
   you have unplugged the USB devices with mounted filesystems on them;
   see the FAQ below for details.  (This is not true for more traditional
   power states like "standby", which normally don't turn USB off.)

Swap partition:
  You need to append resume=/dev/your_swap_partition to kernel command
  line or specify it using /sys/power/resume.

Swap file:
  If using a swapfile you can also specify a resume offset using
  resume_offset=<number> on the kernel command line or specify it
  in /sys/power/resume_offset.

After preparing then you suspend by::

	echo shutdown > /sys/power/disk; echo disk > /sys/power/state

- If you feel ACPI works pretty well on your system, you might try::

	echo platform > /sys/power/disk; echo disk > /sys/power/state

- If you would like to write hibernation image to swap and then suspend
  to RAM (provided your platform supports it), you can try::

	echo suspend > /sys/power/disk; echo disk > /sys/power/state

- If you have SATA disks, you'll need recent kernels with SATA suspend
  support. For suspend and resume to work, make sure your disk drivers
  are built into kernel -- not modules. [There's way to make
  suspend/resume with modular disk drivers, see FAQ, but you probably
  should not do that.]

If you want to limit the suspend image size to N bytes, do::

	echo N > /sys/power/image_size

before suspend (it is limited to around 2/5 of available RAM by default).

- The resume process checks for the presence of the resume device,
  if found, it then checks the contents for the hibernation image signature.
  If both are found, it resumes the hibernation image.

- The resume process may be triggered in two ways:

  1) During lateinit:  If resume=/dev/your_swap_partition is specified on
     the kernel command line, lateinit runs the resume process.  If the
     resume device has not been probed yet, the resume process fails and
     bootup continues.
  2) Manually from an initrd or initramfs:  May be run from
     the init script by using the /sys/power/resume file.  It is vital
     that this be done prior to remounting any filesystems (even as
     read-only) otherwise data may be corrupted.

Article about goals and implementation of Software Suspend for Linux
====================================================================

Author: Gábor Kuti
Last revised: 2003-10-20 by Pavel Machek

Idea and goals to achieve
-------------------------

Nowadays it is common in several laptops that they have a suspend button. It
saves the state of the machine to a filesystem or to a partition and switches
to standby mode. Later resuming the machine the saved state is loaded back to
ram and the machine can continue its work. It has two real benefits. First we
save ourselves the time machine goes down and later boots up, energy costs
are real high when running from batteries. The other gain is that we don't have
to interrupt our programs so processes that are calculating something for a long
time shouldn't need to be written interruptible.

swsusp saves the state of the machine into active swaps and then reboots or
powerdowns.  You must explicitly specify the swap partition to resume from with
`resume=` kernel option. If signature is found it loads and restores saved
state. If the option `noresume` is specified as a boot parameter, it skips
the resuming.  If the option `hibernate=nocompress` is specified as a boot
parameter, it saves hibernation image without compression.

In the meantime while the system is suspended you should not add/remove any
of the hardware, write to the filesystems, etc.

Sleep states summary
====================

There are three different interfaces you can use, /proc/acpi should
work like this:

In a really perfect world::

  echo 1 > /proc/acpi/sleep       # for standby
  echo 2 > /proc/acpi/sleep       # for suspend to ram
  echo 3 > /proc/acpi/sleep       # for suspend to ram, but with more power
                                  # conservative
  echo 4 > /proc/acpi/sleep       # for suspend to disk
  echo 5 > /proc/acpi/sleep       # for shutdown unfriendly the system

and perhaps::

  echo 4b > /proc/acpi/sleep      # for suspend to disk via s4bios

Frequently Asked Questions
==========================

Q:
  well, suspending a server is IMHO a really stupid thing,
  but... (Diego Zuccato):

A:
  You bought new UPS for your server. How do you install it without
  bringing machine down? Suspend to disk, rearrange power cables,
  resume.

  You have your server on UPS. Power died, and UPS is indicating 30
  seconds to failure. What do you do? Suspend to disk.

Q:
  Maybe I'm missing something, but why don't the regular I/O paths work?

A:
  We do use the regular I/O paths. However we cannot restore the data
  to its original location as we load it. That would create an
  inconsistent kernel state which would certainly result in an oops.
  Instead, we load the image into unused memory and then atomically copy
  it back to it original location. This implies, of course, a maximum
  image size of half the amount of memory.

  There are two solutions to this:

  * require half of memory to be free during suspend. That way you can
    read "new" data onto free spots, then cli and copy

  * assume we had special "polling" ide driver that only uses memory
    between 0-640KB. That way, I'd have to make sure that 0-640KB is free
    during suspending, but otherwise it would work...

  suspend2 shares this fundamental limitation, but does not include user
  data and disk caches into "used memory" by saving them in
  advance. That means that the limitation goes away in practice.

Q:
  Does linux support ACPI S4?

A:
  Yes. That's what echo platform > /sys/power/disk does.

Q:
  What is 'suspend2'?

A:
  suspend2 is 'Software Suspend 2', a forked implementation of
  suspend-to-disk which is available as separate patches for 2.4 and 2.6
  kernels from swsusp.sourceforge.net. It includes support for SMP, 4GB
  highmem and preemption. It also has a extensible architecture that
  allows for arbitrary transformations on the image (compression,
  encryption) and arbitrary backends for writing the image (eg to swap
  or an NFS share[Work In Progress]). Questions regarding suspend2
  should be sent to the mailing list available through the suspend2
  website, and not to the Linux Kernel Mailing List. We are working
  toward merging suspend2 into the mainline kernel.

Q:
  What is the freezing of tasks and why are we using it?

A:
  The freezing of tasks is a mechanism by which user space processes and some
  kernel threads are controlled during hibernation or system-wide suspend (on
  some architectures).  See freezing-of-tasks.txt for details.

Q:
  What is the difference between "platform" and "shutdown"?

A:
  shutdown:
	save state in linux, then tell bios to powerdown

  platform:
	save state in linux, then tell bios to powerdown and blink
        "suspended led"

  "platform" is actually right thing to do where supported, but
  "shutdown" is most reliable (except on ACPI systems).

Q:
  I do not understand why you have such strong objections to idea of
  selective suspend.

A:
  Do selective suspend during runtime power management, that's okay. But
  it's useless for suspend-to-disk. (And I do not see how you could use
  it for suspend-to-ram, I hope you do not want that).

  Lets see, so you suggest to

  * SUSPEND all but swap device and parents
  * Snapshot
  * Write image to disk
  * SUSPEND swap device and parents
  * Powerdown

  Oh no, that does not work, if swap device or its parents uses DMA,
  you've corrupted data. You'd have to do

  * SUSPEND all but swap device and parents
  * FREEZE swap device and parents
  * Snapshot
  * UNFREEZE swap device and parents
  * Write
  * SUSPEND swap device and parents

  Which means that you still need that FREEZE state, and you get more
  complicated code. (And I have not yet introduce details like system
  devices).

Q:
  There don't seem to be any generally useful behavioral
  distinctions between SUSPEND and FREEZE.

A:
  Doing SUSPEND when you are asked to do FREEZE is always correct,
  but it may be unnecessarily slow. If you want your driver to stay simple,
  slowness may not matter to you. It can always be fixed later.

  For devices like disk it does matter, you do not want to spindown for
  FREEZE.

Q:
  After resuming, system is paging heavily, leading to very bad interactivity.

A:
  Try running::

    cat /proc/[0-9]*/maps | grep / | sed 's:.* /:/:' | sort -u | while read file
    do
      test -f "$file" && cat "$file" > /dev/null
    done

  after resume. swapoff -a; swapon -a may also be useful.

Q:
  What happens to devices during swsusp? They seem to be resumed
  during system suspend?

A:
  That's correct. We need to resume them if we want to write image to
  disk. Whole sequence goes like

      **Suspend part**

      running system, user asks for suspend-to-disk

      user processes are stopped

      suspend(PMSG_FREEZE): devices are frozen so that they don't interfere
      with state snapshot

      state snapshot: copy of whole used memory is taken with interrupts
      disabled

      resume(): devices are woken up so that we can write image to swap

      write image to swap

      suspend(PMSG_SUSPEND): suspend devices so that we can power off

      turn the power off

      **Resume part**

      (is actually pretty similar)

      running system, user asks for suspend-to-disk

      user processes are stopped (in common case there are none,
      but with resume-from-initrd, no one knows)

      read image from disk

      suspend(PMSG_FREEZE): devices are frozen so that they don't interfere
      with image restoration

      image restoration: rewrite memory with image

      resume(): devices are woken up so that system can continue

      thaw all user processes

Q:
  What is this 'Encrypt suspend image' for?

A:
  First of all: it is not a replacement for dm-crypt encrypted swap.
  It cannot protect your computer while it is suspended. Instead it does
  protect from leaking sensitive data after resume from suspend.

  Think of the following: you suspend while an application is running
  that keeps sensitive data in memory. The application itself prevents
  the data from being swapped out. Suspend, however, must write these
  data to swap to be able to resume later on. Without suspend encryption
  your sensitive data are then stored in plaintext on disk.  This means
  that after resume your sensitive data are accessible to all
  applications having direct access to the swap device which was used
  for suspend. If you don't need swap after resume these data can remain
  on disk virtually forever. Thus it can happen that your system gets
  broken in weeks later and sensitive data which you thought were
  encrypted and protected are retrieved and stolen from the swap device.
  To prevent this situation you should use 'Encrypt suspend image'.

  During suspend a temporary key is created and this key is used to
  encrypt the data written to disk. When, during resume, the data was
  read back into memory the temporary key is destroyed which simply
  means that all data written to disk during suspend are then
  inaccessible so they can't be stolen later on.  The only thing that
  you must then take care of is that you call 'mkswap' for the swap
  partition used for suspend as early as possible during regular
  boot. This asserts that any temporary key from an oopsed suspend or
  from a failed or aborted resume is erased from the swap device.

  As a rule of thumb use encrypted swap to protect your data while your
  system is shut down or suspended. Additionally use the encrypted
  suspend image to prevent sensitive data from being stolen after
  resume.

Q:
  Can I suspend to a swap file?

A:
  Generally, yes, you can.  However, it requires you to use the "resume=" and
  "resume_offset=" kernel command line parameters, so the resume from a swap
  file cannot be initiated from an initrd or initramfs image.  See
  swsusp-and-swap-files.txt for details.

Q:
  Is there a maximum system RAM size that is supported by swsusp?

A:
  It should work okay with highmem.

Q:
  Does swsusp (to disk) use only one swap partition or can it use
  multiple swap partitions (aggregate them into one logical space)?

A:
  Only one swap partition, sorry.

Q:
  If my application(s) causes lots of memory & swap space to be used
  (over half of the total system RAM), is it correct that it is likely
  to be useless to try to suspend to disk while that app is running?

A:
  No, it should work okay, as long as your app does not mlock()
  it. Just prepare big enough swap partition.

Q:
  What information is useful for debugging suspend-to-disk problems?

A:
  Well, last messages on the screen are always useful. If something
  is broken, it is usually some kernel driver, therefore trying with as
  little as possible modules loaded helps a lot. I also prefer people to
  suspend from console, preferably without X running. Booting with
  init=/bin/bash, then swapon and starting suspend sequence manually
  usually does the trick. Then it is good idea to try with latest
  vanilla kernel.

Q:
  How can distributions ship a swsusp-supporting kernel with modular
  disk drivers (especially SATA)?

A:
  Well, it can be done, load the drivers, then do echo into
  /sys/power/resume file from initrd. Be sure not to mount
  anything, not even read-only mount, or you are going to lose your
  data.

Q:
  How do I make suspend more verbose?

A:
  If you want to see any non-error kernel messages on the virtual
  terminal the kernel switches to during suspend, you have to set the
  kernel console loglevel to at least 4 (KERN_WARNING), for example by
  doing::

	# save the old loglevel
	read LOGLEVEL DUMMY < /proc/sys/kernel/printk
	# set the loglevel so we see the progress bar.
	# if the level is higher than needed, we leave it alone.
	if [ $LOGLEVEL -lt 5 ]; then
	        echo 5 > /proc/sys/kernel/printk
		fi

        IMG_SZ=0
        read IMG_SZ < /sys/power/image_size
        echo -n disk > /sys/power/state
        RET=$?
        #
        # the logic here is:
        # if image_size > 0 (without kernel support, IMG_SZ will be zero),
        # then try again with image_size set to zero.
	if [ $RET -ne 0 -a $IMG_SZ -ne 0 ]; then # try again with minimal image size
                echo 0 > /sys/power/image_size
                echo -n disk > /sys/power/state
                RET=$?
        fi

	# restore previous loglevel
	echo $LOGLEVEL > /proc/sys/kernel/printk
	exit $RET

Q:
  Is this true that if I have a mounted filesystem on a USB device and
  I suspend to disk, I can lose data unless the filesystem has been mounted
  with "sync"?

A:
  That's right ... if you disconnect that device, you may lose data.
  In fact, even with "-o sync" you can lose data if your programs have
  information in buffers they haven't written out to a disk you disconnect,
  or if you disconnect before the device finished saving data you wrote.

  Software suspend normally powers down USB controllers, which is equivalent
  to disconnecting all USB devices attached to your system.

  Your system might well support low-power modes for its USB controllers
  while the system is asleep, maintaining the connection, using true sleep
  modes like "suspend-to-RAM" or "standby".  (Don't write "disk" to the
  /sys/power/state file; write "standby" or "mem".)  We've not seen any
  hardware that can use these modes through software suspend, although in
  theory some systems might support "platform" modes that won't break the
  USB connections.

  Remember that it's always a bad idea to unplug a disk drive containing a
  mounted filesystem.  That's true even when your system is asleep!  The
  safest thing is to unmount all filesystems on removable media (such USB,
  Firewire, CompactFlash, MMC, external SATA, or even IDE hotplug bays)
  before suspending; then remount them after resuming.

  There is a work-around for this problem.  For more information, see
  Documentation/driver-api/usb/persist.rst.

Q:
  Can I suspend-to-disk using a swap partition under LVM?

A:
  Yes and No.  You can suspend successfully, but the kernel will not be able
  to resume on its own.  You need an initramfs that can recognize the resume
  situation, activate the logical volume containing the swap volume (but not
  touch any filesystems!), and eventually call::

    echo -n "$major:$minor" > /sys/power/resume

  where $major and $minor are the respective major and minor device numbers of
  the swap volume.

  uswsusp works with LVM, too.  See http://suspend.sourceforge.net/

Q:
  I upgraded the kernel from 2.6.15 to 2.6.16. Both kernels were
  compiled with the similar configuration files. Anyway I found that
  suspend to disk (and resume) is much slower on 2.6.16 compared to
  2.6.15. Any idea for why that might happen or how can I speed it up?

A:
  This is because the size of the suspend image is now greater than
  for 2.6.15 (by saving more data we can get more responsive system
  after resume).

  There's the /sys/power/image_size knob that controls the size of the
  image.  If you set it to 0 (eg. by echo 0 > /sys/power/image_size as
  root), the 2.6.15 behavior should be restored.  If it is still too
  slow, take a look at suspend.sf.net -- userland suspend is faster and
  supports LZF compression to speed it up further.

repo_name: angery/linux


code:
================
swsusp/S3 tricks
================

Pavel Machek <pavel@ucw.cz>

If you want to trick swsusp/S3 into working, you might want to try:

* go with minimal config, turn off drivers like USB, AGP you don't
  really need

* turn off APIC and preempt

* use ext2. At least it has working fsck. [If something seems to go
  wrong, force fsck when you have a chance]

* turn off modules

* use vga text console, shut down X. [If you really want X, you might
  want to try vesafb later]

* try running as few processes as possible, preferably go to single
  user mode.

* due to video issues, swsusp should be easier to get working than
  S3. Try that first.

When you make it work, try to find out what exactly was it that broke
suspend, and preferably fix that.


code:
=====================================================
Documentation for userland software suspend interface
=====================================================

	(C) 2006 Rafael J. Wysocki <rjw@sisk.pl>

First, the warnings at the beginning of swsusp.txt still apply.

Second, you should read the FAQ in swsusp.txt _now_ if you have not
done it already.

Now, to use the userland interface for software suspend you need special
utilities that will read/write the system memory snapshot from/to the
kernel.  Such utilities are available, for example, from
<http://suspend.sourceforge.net>.  You may want to have a look at them if you
are going to develop your own suspend/resume utilities.

The interface consists of a character device providing the open(),
release(), read(), and write() operations as well as several ioctl()
commands defined in include/linux/suspend_ioctls.h .  The major and minor
numbers of the device are, respectively, 10 and 231, and they can
be read from /sys/class/misc/snapshot/dev.

The device can be open either for reading or for writing.  If open for
reading, it is considered to be in the suspend mode.  Otherwise it is
assumed to be in the resume mode.  The device cannot be open for simultaneous
reading and writing.  It is also impossible to have the device open more than
once at a time.

Even opening the device has side effects. Data structures are
allocated, and PM_HIBERNATION_PREPARE / PM_RESTORE_PREPARE chains are
called.

The ioctl() commands recognized by the device are:

SNAPSHOT_FREEZE
	freeze user space processes (the current process is
	not frozen); this is required for SNAPSHOT_CREATE_IMAGE
	and SNAPSHOT_ATOMIC_RESTORE to succeed

SNAPSHOT_UNFREEZE
	thaw user space processes frozen by SNAPSHOT_FREEZE

SNAPSHOT_CREATE_IMAGE
	create a snapshot of the system memory; the
	last argument of ioctl() should be a pointer to an int variable,
	the value of which will indicate whether the call returned after
	creating the snapshot (1) or after restoring the system memory state
	from it (0) (after resume the system finds itself finishing the
	SNAPSHOT_CREATE_IMAGE ioctl() again); after the snapshot
	has been created the read() operation can be used to transfer
	it out of the kernel

SNAPSHOT_ATOMIC_RESTORE
	restore the system memory state from the
	uploaded snapshot image; before calling it you should transfer
	the system memory snapshot back to the kernel using the write()
	operation; this call will not succeed if the snapshot
	image is not available to the kernel

SNAPSHOT_FREE
	free memory allocated for the snapshot image

SNAPSHOT_PREF_IMAGE_SIZE
	set the preferred maximum size of the image
	(the kernel will do its best to ensure the image size will not exceed
	this number, but if it turns out to be impossible, the kernel will
	create the smallest image possible)

SNAPSHOT_GET_IMAGE_SIZE
	return the actual size of the hibernation image
	(the last argument should be a pointer to a loff_t variable that
	will contain the result if the call is successful)

SNAPSHOT_AVAIL_SWAP_SIZE
	return the amount of available swap in bytes
	(the last argument should be a pointer to a loff_t variable that
	will contain the result if the call is successful)

SNAPSHOT_ALLOC_SWAP_PAGE
	allocate a swap page from the resume partition
	(the last argument should be a pointer to a loff_t variable that
	will contain the swap page offset if the call is successful)

SNAPSHOT_FREE_SWAP_PAGES
	free all swap pages allocated by
	SNAPSHOT_ALLOC_SWAP_PAGE

SNAPSHOT_SET_SWAP_AREA
	set the resume partition and the offset (in <PAGE_SIZE>
	units) from the beginning of the partition at which the swap header is
	located (the last ioctl() argument should point to a struct
	resume_swap_area, as defined in kernel/power/suspend_ioctls.h,
	containing the resume device specification and the offset); for swap
	partitions the offset is always 0, but it is different from zero for
	swap files (see Documentation/power/swsusp-and-swap-files.rst for
	details).

SNAPSHOT_PLATFORM_SUPPORT
	enable/disable the hibernation platform support,
	depending on the argument value (enable, if the argument is nonzero)

SNAPSHOT_POWER_OFF
	make the kernel transition the system to the hibernation
	state (eg. ACPI S4) using the platform (eg. ACPI) driver

SNAPSHOT_S2RAM
	suspend to RAM; using this call causes the kernel to
	immediately enter the suspend-to-RAM state, so this call must always
	be preceded by the SNAPSHOT_FREEZE call and it is also necessary
	to use the SNAPSHOT_UNFREEZE call after the system wakes up.  This call
	is needed to implement the suspend-to-both mechanism in which the
	suspend image is first created, as though the system had been suspended
	to disk, and then the system is suspended to RAM (this makes it possible
	to resume the system from RAM if there's enough battery power or restore
	its state on the basis of the saved suspend image otherwise)

The device's read() operation can be used to transfer the snapshot image from
the kernel.  It has the following limitations:

- you cannot read() more than one virtual memory page at a time
- read()s across page boundaries are impossible (ie. if you read() 1/2 of
  a page in the previous call, you will only be able to read()
  **at most** 1/2 of the page in the next call)

The device's write() operation is used for uploading the system memory snapshot
into the kernel.  It has the same limitations as the read() operation.

The release() operation frees all memory allocated for the snapshot image
and all swap pages allocated with SNAPSHOT_ALLOC_SWAP_PAGE (if any).
Thus it is not necessary to use either SNAPSHOT_FREE or
SNAPSHOT_FREE_SWAP_PAGES before closing the device (in fact it will also
unfreeze user space processes frozen by SNAPSHOT_UNFREEZE if they are
still frozen when the device is being closed).

Currently it is assumed that the userland utilities reading/writing the
snapshot image from/to the kernel will use a swap partition, called the resume
partition, or a swap file as storage space (if a swap file is used, the resume
partition is the partition that holds this file).  However, this is not really
required, as they can use, for example, a special (blank) suspend partition or
a file on a partition that is unmounted before SNAPSHOT_CREATE_IMAGE and
mounted afterwards.

These utilities MUST NOT make any assumptions regarding the ordering of
data within the snapshot image.  The contents of the image are entirely owned
by the kernel and its structure may be changed in future kernel releases.

The snapshot image MUST be written to the kernel unaltered (ie. all of the image
data, metadata and header MUST be written in _exactly_ the same amount, form
and order in which they have been read).  Otherwise, the behavior of the
resumed system may be totally unpredictable.

While executing SNAPSHOT_ATOMIC_RESTORE the kernel checks if the
structure of the snapshot image is consistent with the information stored
in the image header.  If any inconsistencies are detected,
SNAPSHOT_ATOMIC_RESTORE will not succeed.  Still, this is not a fool-proof
mechanism and the userland utilities using the interface SHOULD use additional
means, such as checksums, to ensure the integrity of the snapshot image.

The suspending and resuming utilities MUST lock themselves in memory,
preferably using mlockall(), before calling SNAPSHOT_FREEZE.

The suspending utility MUST check the value stored by SNAPSHOT_CREATE_IMAGE
in the memory location pointed to by the last argument of ioctl() and proceed
in accordance with it:

1. 	If the value is 1 (ie. the system memory snapshot has just been
	created and the system is ready for saving it):

	(a)	The suspending utility MUST NOT close the snapshot device
		_unless_ the whole suspend procedure is to be cancelled, in
		which case, if the snapshot image has already been saved, the
		suspending utility SHOULD destroy it, preferably by zapping
		its header.  If the suspend is not to be cancelled, the
		system MUST be powered off or rebooted after the snapshot
		image has been saved.
	(b)	The suspending utility SHOULD NOT attempt to perform any
		file system operations (including reads) on the file systems
		that were mounted before SNAPSHOT_CREATE_IMAGE has been
		called.  However, it MAY mount a file system that was not
		mounted at that time and perform some operations on it (eg.
		use it for saving the image).

2.	If the value is 0 (ie. the system state has just been restored from
	the snapshot image), the suspending utility MUST close the snapshot
	device.  Afterwards it will be treated as a regular userland process,
	so it need not exit.

The resuming utility SHOULD NOT attempt to mount any file systems that could
be mounted before suspend and SHOULD NOT attempt to perform any operations
involving such file systems.

For details, please refer to the source code.


code:
===========================
Video issues with S3 resume
===========================

2003-2006, Pavel Machek

During S3 resume, hardware needs to be reinitialized. For most
devices, this is easy, and kernel driver knows how to do
it. Unfortunately there's one exception: video card. Those are usually
initialized by BIOS, and kernel does not have enough information to
boot video card. (Kernel usually does not even contain video card
driver -- vesafb and vgacon are widely used).

This is not problem for swsusp, because during swsusp resume, BIOS is
run normally so video card is normally initialized. It should not be
problem for S1 standby, because hardware should retain its state over
that.

We either have to run video BIOS during early resume, or interpret it
using vbetool later, or maybe nothing is necessary on particular
system because video state is preserved. Unfortunately different
methods work on different systems, and no known method suits all of
them.

Userland application called s2ram has been developed; it contains long
whitelist of systems, and automatically selects working method for a
given system. It can be downloaded from CVS at
www.sf.net/projects/suspend . If you get a system that is not in the
whitelist, please try to find a working solution, and submit whitelist
entry so that work does not need to be repeated.

Currently, VBE_SAVE method (6 below) works on most
systems. Unfortunately, vbetool only runs after userland is resumed,
so it makes debugging of early resume problems
hard/impossible. Methods that do not rely on userland are preferable.

Details
~~~~~~~

There are a few types of systems where video works after S3 resume:

(1) systems where video state is preserved over S3.

(2) systems where it is possible to call the video BIOS during S3
    resume. Unfortunately, it is not correct to call the video BIOS at
    that point, but it happens to work on some machines. Use
    acpi_sleep=s3_bios.

(3) systems that initialize video card into vga text mode and where
    the BIOS works well enough to be able to set video mode. Use
    acpi_sleep=s3_mode on these.

(4) on some systems s3_bios kicks video into text mode, and
    acpi_sleep=s3_bios,s3_mode is needed.

(5) radeon systems, where X can soft-boot your video card. You'll need
    a new enough X, and a plain text console (no vesafb or radeonfb). See
    http://www.doesi.gmxhome.de/linux/tm800s3/s3.html for more information.
    Alternatively, you should use vbetool (6) instead.

(6) other radeon systems, where vbetool is enough to bring system back
    to life. It needs text console to be working. Do vbetool vbestate
    save > /tmp/delme; echo 3 > /proc/acpi/sleep; vbetool post; vbetool
    vbestate restore < /tmp/delme; setfont <whatever>, and your video
    should work.

(7) on some systems, it is possible to boot most of kernel, and then
    POSTing bios works. Ole Rohne has patch to do just that at
    http://dev.gentoo.org/~marineam/patch-radeonfb-2.6.11-rc2-mm2.

(8) on some systems, you can use the video_post utility and or
    do echo 3 > /sys/power/state  && /usr/sbin/video_post - which will
    initialize the display in console mode. If you are in X, you can switch
    to a virtual terminal and back to X using  CTRL+ALT+F1 - CTRL+ALT+F7 to get
    the display working in graphical mode again.

Now, if you pass acpi_sleep=something, and it does not work with your
bios, you'll get a hard crash during resume. Be careful. Also it is
safest to do your experiments with plain old VGA console. The vesafb
and radeonfb (etc) drivers have a tendency to crash the machine during
resume.

You may have a system where none of above works. At that point you
either invent another ugly hack that works, or write proper driver for
your video card (good luck getting docs :-(). Maybe suspending from X
(proper X, knowing your hardware, not XF68_FBcon) might have better
chance of working.

Table of known working notebooks:

=============================== ===============================================
Model                           hack (or "how to do it")
=============================== ===============================================
Acer Aspire 1406LC		ole's late BIOS init (7), turn off DRI
Acer TM 230			s3_bios (2)
Acer TM 242FX			vbetool (6)
Acer TM C110			video_post (8)
Acer TM C300                    vga=normal (only suspend on console, not in X),
				vbetool (6) or video_post (8)
Acer TM 4052LCi		        s3_bios (2)
Acer TM 636Lci			s3_bios,s3_mode (4)
Acer TM 650 (Radeon M7)		vga=normal plus boot-radeon (5) gets text
				console back
Acer TM 660			??? [#f1]_
Acer TM 800			vga=normal, X patches, see webpage (5)
				or vbetool (6)
Acer TM 803			vga=normal, X patches, see webpage (5)
				or vbetool (6)
Acer TM 803LCi			vga=normal, vbetool (6)
Arima W730a			vbetool needed (6)
Asus L2400D                     s3_mode (3) [#f2]_ (S1 also works OK)
Asus L3350M (SiS 740)           (6)
Asus L3800C (Radeon M7)		s3_bios (2) (S1 also works OK)
Asus M6887Ne			vga=normal, s3_bios (2), use radeon driver
				instead of fglrx in x.org
Athlon64 desktop prototype	s3_bios (2)
Compal CL-50			??? [#f1]_
Compaq Armada E500 - P3-700     none (1) (S1 also works OK)
Compaq Evo N620c		vga=normal, s3_bios (2)
Dell 600m, ATI R250 Lf		none (1), but needs xorg-x11-6.8.1.902-1
Dell D600, ATI RV250            vga=normal and X, or try vbestate (6)
Dell D610			vga=normal and X (possibly vbestate (6) too,
				but not tested)
Dell Inspiron 4000		??? [#f1]_
Dell Inspiron 500m		??? [#f1]_
Dell Inspiron 510m		???
Dell Inspiron 5150		vbetool needed (6)
Dell Inspiron 600m		??? [#f1]_
Dell Inspiron 8200		??? [#f1]_
Dell Inspiron 8500		??? [#f1]_
Dell Inspiron 8600		??? [#f1]_
eMachines athlon64 machines	vbetool needed (6) (someone please get
				me model #s)
HP NC6000			s3_bios, may not use radeonfb (2);
				or vbetool (6)
HP NX7000			??? [#f1]_
HP Pavilion ZD7000		vbetool post needed, need open-source nv
				driver for X
HP Omnibook XE3	athlon version	none (1)
HP Omnibook XE3GC		none (1), video is S3 Savage/IX-MV
HP Omnibook XE3L-GF		vbetool (6)
HP Omnibook 5150		none (1), (S1 also works OK)
IBM TP T20, model 2647-44G	none (1), video is S3 Inc. 86C270-294
				Savage/IX-MV, vesafb gets "interesting"
				but X work.
IBM TP A31 / Type 2652-M5G      s3_mode (3) [works ok with
				BIOS 1.04 2002-08-23, but not at all with
				BIOS 1.11 2004-11-05 :-(]
IBM TP R32 / Type 2658-MMG      none (1)
IBM TP R40 2722B3G		??? [#f1]_
IBM TP R50p / Type 1832-22U     s3_bios (2)
IBM TP R51			none (1)
IBM TP T30	236681A		??? [#f1]_
IBM TP T40 / Type 2373-MU4      none (1)
IBM TP T40p			none (1)
IBM TP R40p			s3_bios (2)
IBM TP T41p			s3_bios (2), switch to X after resume
IBM TP T42			s3_bios (2)
IBM ThinkPad T42p (2373-GTG)	s3_bios (2)
IBM TP X20			??? [#f1]_
IBM TP X30			s3_bios, s3_mode (4)
IBM TP X31 / Type 2672-XXH      none (1), use radeontool
				(http://fdd.com/software/radeon/) to
				turn off backlight.
IBM TP X32			none (1), but backlight is on and video is
				trashed after long suspend. s3_bios,
				s3_mode (4) works too. Perhaps that gets
				better results?
IBM Thinkpad X40 Type 2371-7JG  s3_bios,s3_mode (4)
IBM TP 600e			none(1), but a switch to console and
				back to X is needed
Medion MD4220			??? [#f1]_
Samsung P35			vbetool needed (6)
Sharp PC-AR10 (ATI rage)	none (1), backlight does not switch off
Sony Vaio PCG-C1VRX/K		s3_bios (2)
Sony Vaio PCG-F403		??? [#f1]_
Sony Vaio PCG-GRT995MP		none (1), works with 'nv' X driver
Sony Vaio PCG-GR7/K		none (1), but needs radeonfb, use
				radeontool (http://fdd.com/software/radeon/)
				to turn off backlight.
Sony Vaio PCG-N505SN		??? [#f1]_
Sony Vaio vgn-s260		X or boot-radeon can init it (5)
Sony Vaio vgn-S580BH		vga=normal, but suspend from X. Console will
				be blank unless you return to X.
Sony Vaio vgn-FS115B		s3_bios (2),s3_mode (4)
Toshiba Libretto L5		none (1)
Toshiba Libretto 100CT/110CT    vbetool (6)
Toshiba Portege 3020CT		s3_mode (3)
Toshiba Satellite 4030CDT	s3_mode (3) (S1 also works OK)
Toshiba Satellite 4080XCDT      s3_mode (3) (S1 also works OK)
Toshiba Satellite 4090XCDT      ??? [#f1]_
Toshiba Satellite P10-554       s3_bios,s3_mode (4)[#f3]_
Toshiba M30                     (2) xor X with nvidia driver using internal AGP
Uniwill 244IIO			??? [#f1]_
=============================== ===============================================

Known working desktop systems
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

=================== ============================= ========================
Mainboard	    Graphics card                 hack (or "how to do it")
=================== ============================= ========================
Asus A7V8X	    nVidia RIVA TNT2 model 64	  s3_bios,s3_mode (4)
=================== ============================= ========================

.. [#f1] from https://wiki.ubuntu.com/HoaryPMResults, not sure
         which options to use. If you know, please tell me.

.. [#f2] To be tested with a newer kernel.

.. [#f3] Not with SMP kernel, UP only.


code:
============================
NUMA resource associativity
============================

Associativity represents the groupings of the various platform resources into
domains of substantially similar mean performance relative to resources outside
of that domain. Resources subsets of a given domain that exhibit better
performance relative to each other than relative to other resources subsets
are represented as being members of a sub-grouping domain. This performance
characteristic is presented in terms of NUMA node distance within the Linux kernel.
From the platform view, these groups are also referred to as domains.

PAPR interface currently supports different ways of communicating these resource
grouping details to the OS. These are referred to as Form 0, Form 1 and Form2
associativity grouping. Form 0 is the oldest format and is now considered deprecated.

Hypervisor indicates the type/form of associativity used via "ibm,architecture-vec-5 property".
Bit 0 of byte 5 in the "ibm,architecture-vec-5" property indicates usage of Form 0 or Form 1.
A value of 1 indicates the usage of Form 1 associativity. For Form 2 associativity
bit 2 of byte 5 in the "ibm,architecture-vec-5" property is used.

Form 0
------
Form 0 associativity supports only two NUMA distances (LOCAL and REMOTE).

Form 1
------
With Form 1 a combination of ibm,associativity-reference-points, and ibm,associativity
device tree properties are used to determine the NUMA distance between resource groups/domains.

The “ibm,associativity” property contains a list of one or more numbers (domainID)
representing the resource’s platform grouping domains.

The “ibm,associativity-reference-points” property contains a list of one or more numbers
(domainID index) that represents the 1 based ordinal in the associativity lists.
The list of domainID indexes represents an increasing hierarchy of resource grouping.

ex:
{ primary domainID index, secondary domainID index, tertiary domainID index.. }

Linux kernel uses the domainID at the primary domainID index as the NUMA node id.
Linux kernel computes NUMA distance between two domains by recursively comparing
if they belong to the same higher-level domains. For mismatch at every higher
level of the resource group, the kernel doubles the NUMA distance between the
comparing domains.

Form 2
-------
Form 2 associativity format adds separate device tree properties representing NUMA node distance
thereby making the node distance computation flexible. Form 2 also allows flexible primary
domain numbering. With numa distance computation now detached from the index value in
"ibm,associativity-reference-points" property, Form 2 allows a large number of primary domain
ids at the same domainID index representing resource groups of different performance/latency
characteristics.

Hypervisor indicates the usage of FORM2 associativity using bit 2 of byte 5 in the
"ibm,architecture-vec-5" property.

"ibm,numa-lookup-index-table" property contains a list of one or more numbers representing
the domainIDs present in the system. The offset of the domainID in this property is
used as an index while computing numa distance information via "ibm,numa-distance-table".

prop-encoded-array: The number N of the domainIDs encoded as with encode-int, followed by
N domainID encoded as with encode-int

For ex:
"ibm,numa-lookup-index-table" =  {4, 0, 8, 250, 252}. The offset of domainID 8 (2) is used when
computing the distance of domain 8 from other domains present in the system. For the rest of
this document, this offset will be referred to as domain distance offset.

"ibm,numa-distance-table" property contains a list of one or more numbers representing the NUMA
distance between resource groups/domains present in the system.

prop-encoded-array: The number N of the distance values encoded as with encode-int, followed by
N distance values encoded as with encode-bytes. The max distance value we could encode is 255.
The number N must be equal to the square of m where m is the number of domainIDs in the
numa-lookup-index-table.

For ex:
ibm,numa-lookup-index-table = <3 0 8 40>;
ibm,numa-distace-table = <9>, /bits/ 8 < 10  20  80 20  10 160 80 160  10>;

::

	  | 0    8   40
	--|------------
	  |
	0 | 10   20  80
	  |
	8 | 20   10  160
	  |
	40| 80   160  10

A possible "ibm,associativity" property for resources in node 0, 8 and 40

{ 3, 6, 7, 0 }
{ 3, 6, 9, 8 }
{ 3, 6, 7, 40}

With "ibm,associativity-reference-points"  { 0x3 }

"ibm,lookup-index-table" helps in having a compact representation of distance matrix.
Since domainID can be sparse, the matrix of distances can also be effectively sparse.
With "ibm,lookup-index-table" we can achieve a compact representation of
distance information.


code:
.. SPDX-License-Identifier: GPL-2.0

DeviceTree Booting
------------------

During the development of the Linux/ppc64 kernel, and more specifically, the
addition of new platform types outside of the old IBM pSeries/iSeries pair, it
was decided to enforce some strict rules regarding the kernel entry and
bootloader <-> kernel interfaces, in order to avoid the degeneration that had
become the ppc32 kernel entry point and the way a new platform should be added
to the kernel. The legacy iSeries platform breaks those rules as it predates
this scheme, but no new board support will be accepted in the main tree that
doesn't follow them properly.  In addition, since the advent of the arch/powerpc
merged architecture for ppc32 and ppc64, new 32-bit platforms and 32-bit
platforms which move into arch/powerpc will be required to use these rules as
well.

The main requirement that will be defined in more detail below is the presence
of a device-tree whose format is defined after Open Firmware specification.
However, in order to make life easier to embedded board vendors, the kernel
doesn't require the device-tree to represent every device in the system and only
requires some nodes and properties to be present. For example, the kernel does
not require you to create a node for every PCI device in the system. It is a
requirement to have a node for PCI host bridges in order to provide interrupt
routing information and memory/IO ranges, among others. It is also recommended
to define nodes for on chip devices and other buses that don't specifically fit
in an existing OF specification. This creates a great flexibility in the way the
kernel can then probe those and match drivers to device, without having to hard
code all sorts of tables. It also makes it more flexible for board vendors to do
minor hardware upgrades without significantly impacting the kernel code or
cluttering it with special cases.

Entry point
~~~~~~~~~~~

There is one single entry point to the kernel, at the start
of the kernel image. That entry point supports two calling
conventions:

        a) Boot from Open Firmware. If your firmware is compatible
        with Open Firmware (IEEE 1275) or provides an OF compatible
        client interface API (support for "interpret" callback of
        forth words isn't required), you can enter the kernel with:

              r5 : OF callback pointer as defined by IEEE 1275
              bindings to powerpc. Only the 32-bit client interface
              is currently supported

              r3, r4 : address & length of an initrd if any or 0

              The MMU is either on or off; the kernel will run the
              trampoline located in arch/powerpc/kernel/prom_init.c to
              extract the device-tree and other information from open
              firmware and build a flattened device-tree as described
              in b). prom_init() will then re-enter the kernel using
              the second method. This trampoline code runs in the
              context of the firmware, which is supposed to handle all
              exceptions during that time.

        b) Direct entry with a flattened device-tree block. This entry
        point is called by a) after the OF trampoline and can also be
        called directly by a bootloader that does not support the Open
        Firmware client interface. It is also used by "kexec" to
        implement "hot" booting of a new kernel from a previous
        running one. This method is what I will describe in more
        details in this document, as method a) is simply standard Open
        Firmware, and thus should be implemented according to the
        various standard documents defining it and its binding to the
        PowerPC platform. The entry point definition then becomes:

                r3 : physical pointer to the device-tree block
                (defined in chapter II) in RAM

                r4 : physical pointer to the kernel itself. This is
                used by the assembly code to properly disable the MMU
                in case you are entering the kernel with MMU enabled
                and a non-1:1 mapping.

                r5 : NULL (as to differentiate with method a)

Note about SMP entry: Either your firmware puts your other
CPUs in some sleep loop or spin loop in ROM where you can get
them out via a soft reset or some other means, in which case
you don't need to care, or you'll have to enter the kernel
with all CPUs. The way to do that with method b) will be
described in a later revision of this document.

Board supports (platforms) are not exclusive config options. An
arbitrary set of board supports can be built in a single kernel
image. The kernel will "know" what set of functions to use for a
given platform based on the content of the device-tree. Thus, you
should:

        a) add your platform support as a _boolean_ option in
        arch/powerpc/Kconfig, following the example of PPC_PSERIES,
        PPC_PMAC and PPC_MAPLE. The latter is probably a good
        example of a board support to start from.

        b) create your main platform file as
        "arch/powerpc/platforms/myplatform/myboard_setup.c" and add it
        to the Makefile under the condition of your ``CONFIG_``
        option. This file will define a structure of type "ppc_md"
        containing the various callbacks that the generic code will
        use to get to your platform specific code

A kernel image may support multiple platforms, but only if the
platforms feature the same core architecture.  A single kernel build
cannot support both configurations with Book E and configurations
with classic Powerpc architectures.


code:
========================
The PowerPC boot wrapper
========================

Copyright (C) Secret Lab Technologies Ltd.

PowerPC image targets compresses and wraps the kernel image (vmlinux) with
a boot wrapper to make it usable by the system firmware.  There is no
standard PowerPC firmware interface, so the boot wrapper is designed to
be adaptable for each kind of image that needs to be built.

The boot wrapper can be found in the arch/powerpc/boot/ directory.  The
Makefile in that directory has targets for all the available image types.
The different image types are used to support all of the various firmware
interfaces found on PowerPC platforms.  OpenFirmware is the most commonly
used firmware type on general purpose PowerPC systems from Apple, IBM and
others.  U-Boot is typically found on embedded PowerPC hardware, but there
are a handful of other firmware implementations which are also popular.  Each
firmware interface requires a different image format.

The boot wrapper is built from the makefile in arch/powerpc/boot/Makefile and
it uses the wrapper script (arch/powerpc/boot/wrapper) to generate target
image.  The details of the build system is discussed in the next section.
Currently, the following image format targets exist:

   ==================== ========================================================
   cuImage.%:		Backwards compatible uImage for older version of
			U-Boot (for versions that don't understand the device
			tree).  This image embeds a device tree blob inside
			the image.  The boot wrapper, kernel and device tree
			are all embedded inside the U-Boot uImage file format
			with boot wrapper code that extracts data from the old
			bd_info structure and loads the data into the device
			tree before jumping into the kernel.

			Because of the series of #ifdefs found in the
			bd_info structure used in the old U-Boot interfaces,
			cuImages are platform specific.  Each specific
			U-Boot platform has a different platform init file
			which populates the embedded device tree with data
			from the platform specific bd_info file.  The platform
			specific cuImage platform init code can be found in
			`arch/powerpc/boot/cuboot.*.c`. Selection of the correct
			cuImage init code for a specific board can be found in
			the wrapper structure.

   dtbImage.%:		Similar to zImage, except device tree blob is embedded
			inside the image instead of provided by firmware.  The
			output image file can be either an elf file or a flat
			binary depending on the platform.

			dtbImages are used on systems which do not have an
			interface for passing a device tree directly.
			dtbImages are similar to simpleImages except that
			dtbImages have platform specific code for extracting
			data from the board firmware, but simpleImages do not
			talk to the firmware at all.

			PlayStation 3 support uses dtbImage.  So do Embedded
			Planet boards using the PlanetCore firmware.  Board
			specific initialization code is typically found in a
			file named arch/powerpc/boot/<platform>.c; but this
			can be overridden by the wrapper script.

   simpleImage.%:	Firmware independent compressed image that does not
			depend on any particular firmware interface and embeds
			a device tree blob.  This image is a flat binary that
			can be loaded to any location in RAM and jumped to.
			Firmware cannot pass any configuration data to the
			kernel with this image type and it depends entirely on
			the embedded device tree for all information.

   treeImage.%;		Image format for used with OpenBIOS firmware found
			on some ppc4xx hardware.  This image embeds a device
			tree blob inside the image.

   uImage:		Native image format used by U-Boot.  The uImage target
			does not add any boot code.  It just wraps a compressed
			vmlinux in the uImage data structure.  This image
			requires a version of U-Boot that is able to pass
			a device tree to the kernel at boot.  If using an older
			version of U-Boot, then you need to use a cuImage
			instead.

   zImage.%:		Image format which does not embed a device tree.
			Used by OpenFirmware and other firmware interfaces
			which are able to supply a device tree.  This image
			expects firmware to provide the device tree at boot.
			Typically, if you have general purpose PowerPC
			hardware then you want this image format.
   ==================== ========================================================

Image types which embed a device tree blob (simpleImage, dtbImage, treeImage,
and cuImage) all generate the device tree blob from a file in the
arch/powerpc/boot/dts/ directory.  The Makefile selects the correct device
tree source based on the name of the target.  Therefore, if the kernel is
built with 'make treeImage.walnut', then the build system will use
arch/powerpc/boot/dts/walnut.dts to build treeImage.walnut.

Two special targets called 'zImage' and 'zImage.initrd' also exist.  These
targets build all the default images as selected by the kernel configuration.
Default images are selected by the boot wrapper Makefile
(arch/powerpc/boot/Makefile) by adding targets to the $image-y variable.  Look
at the Makefile to see which default image targets are available.

How it is built
---------------
arch/powerpc is designed to support multiplatform kernels, which means
that a single vmlinux image can be booted on many different target boards.
It also means that the boot wrapper must be able to wrap for many kinds of
images on a single build.  The design decision was made to not use any
conditional compilation code (#ifdef, etc) in the boot wrapper source code.
All of the boot wrapper pieces are buildable at any time regardless of the
kernel configuration.  Building all the wrapper bits on every kernel build
also ensures that obscure parts of the wrapper are at the very least compile
tested in a large variety of environments.

The wrapper is adapted for different image types at link time by linking in
just the wrapper bits that are appropriate for the image type.  The 'wrapper
script' (found in arch/powerpc/boot/wrapper) is called by the Makefile and
is responsible for selecting the correct wrapper bits for the image type.
The arguments are well documented in the script's comment block, so they
are not repeated here.  However, it is worth mentioning that the script
uses the -p (platform) argument as the main method of deciding which wrapper
bits to compile in.  Look for the large 'case "$platform" in' block in the
middle of the script.  This is also the place where platform specific fixups
can be selected by changing the link order.

In particular, care should be taken when working with cuImages.  cuImage
wrapper bits are very board specific and care should be taken to make sure
the target you are trying to build is supported by the wrapper bits.


code:
============
CPU Features
============

Hollis Blanchard <hollis@austin.ibm.com>
5 Jun 2002

This document describes the system (including self-modifying code) used in the
PPC Linux kernel to support a variety of PowerPC CPUs without requiring
compile-time selection.

Early in the boot process the ppc32 kernel detects the current CPU type and
chooses a set of features accordingly. Some examples include Altivec support,
split instruction and data caches, and if the CPU supports the DOZE and NAP
sleep modes.

Detection of the feature set is simple. A list of processors can be found in
arch/powerpc/kernel/cputable.c. The PVR register is masked and compared with
each value in the list. If a match is found, the cpu_features of cur_cpu_spec
is assigned to the feature bitmask for this processor and a __setup_cpu
function is called.

C code may test 'cur_cpu_spec[smp_processor_id()]->cpu_features' for a
particular feature bit. This is done in quite a few places, for example
in ppc_setup_l2cr().

Implementing cpufeatures in assembly is a little more involved. There are
several paths that are performance-critical and would suffer if an array
index, structure dereference, and conditional branch were added. To avoid the
performance penalty but still allow for runtime (rather than compile-time) CPU
selection, unused code is replaced by 'nop' instructions. This nop'ing is
based on CPU 0's capabilities, so a multi-processor system with non-identical
processors will not work (but such a system would likely have other problems
anyways).

After detecting the processor type, the kernel patches out sections of code
that shouldn't be used by writing nop's over it. Using cpufeatures requires
just 2 macros (found in arch/powerpc/include/asm/cputable.h), as seen in head.S
transfer_to_handler::

	#ifdef CONFIG_ALTIVEC
	BEGIN_FTR_SECTION
		mfspr	r22,SPRN_VRSAVE		/* if G4, save vrsave register value */
		stw	r22,THREAD_VRSAVE(r23)
	END_FTR_SECTION_IFSET(CPU_FTR_ALTIVEC)
	#endif /* CONFIG_ALTIVEC */

If CPU 0 supports Altivec, the code is left untouched. If it doesn't, both
instructions are replaced with nop's.

The END_FTR_SECTION macro has two simpler variations: END_FTR_SECTION_IFSET
and END_FTR_SECTION_IFCLR. These simply test if a flag is set (in
cur_cpu_spec[0]->cpu_features) or is cleared, respectively. These two macros
should be used in the majority of cases.

The END_FTR_SECTION macros are implemented by storing information about this
code in the '__ftr_fixup' ELF section. When do_cpu_ftr_fixups
(arch/powerpc/kernel/misc.S) is invoked, it will iterate over the records in
__ftr_fixup, and if the required feature is not present it will loop writing
nop's from each BEGIN_FTR_SECTION to END_FTR_SECTION.


code:
====================================
Coherent Accelerator Interface (CXL)
====================================

Introduction
============

    The coherent accelerator interface is designed to allow the
    coherent connection of accelerators (FPGAs and other devices) to a
    POWER system. These devices need to adhere to the Coherent
    Accelerator Interface Architecture (CAIA).

    IBM refers to this as the Coherent Accelerator Processor Interface
    or CAPI. In the kernel it's referred to by the name CXL to avoid
    confusion with the ISDN CAPI subsystem.

    Coherent in this context means that the accelerator and CPUs can
    both access system memory directly and with the same effective
    addresses.

Hardware overview
=================

    ::

         POWER8/9             FPGA
       +----------+        +---------+
       |          |        |         |
       |   CPU    |        |   AFU   |
       |          |        |         |
       |          |        |         |
       |          |        |         |
       +----------+        +---------+
       |   PHB    |        |         |
       |   +------+        |   PSL   |
       |   | CAPP |<------>|         |
       +---+------+  PCIE  +---------+

    The POWER8/9 chip has a Coherently Attached Processor Proxy (CAPP)
    unit which is part of the PCIe Host Bridge (PHB). This is managed
    by Linux by calls into OPAL. Linux doesn't directly program the
    CAPP.

    The FPGA (or coherently attached device) consists of two parts.
    The POWER Service Layer (PSL) and the Accelerator Function Unit
    (AFU). The AFU is used to implement specific functionality behind
    the PSL. The PSL, among other things, provides memory address
    translation services to allow each AFU direct access to userspace
    memory.

    The AFU is the core part of the accelerator (eg. the compression,
    crypto etc function). The kernel has no knowledge of the function
    of the AFU. Only userspace interacts directly with the AFU.

    The PSL provides the translation and interrupt services that the
    AFU needs. This is what the kernel interacts with. For example, if
    the AFU needs to read a particular effective address, it sends
    that address to the PSL, the PSL then translates it, fetches the
    data from memory and returns it to the AFU. If the PSL has a
    translation miss, it interrupts the kernel and the kernel services
    the fault. The context to which this fault is serviced is based on
    who owns that acceleration function.

    - POWER8 and PSL Version 8 are compliant to the CAIA Version 1.0.
    - POWER9 and PSL Version 9 are compliant to the CAIA Version 2.0.

    This PSL Version 9 provides new features such as:

    * Interaction with the nest MMU on the P9 chip.
    * Native DMA support.
    * Supports sending ASB_Notify messages for host thread wakeup.
    * Supports Atomic operations.
    * etc.

    Cards with a PSL9 won't work on a POWER8 system and cards with a
    PSL8 won't work on a POWER9 system.

AFU Modes
=========

    There are two programming modes supported by the AFU. Dedicated
    and AFU directed. AFU may support one or both modes.

    When using dedicated mode only one MMU context is supported. In
    this mode, only one userspace process can use the accelerator at
    time.

    When using AFU directed mode, up to 16K simultaneous contexts can
    be supported. This means up to 16K simultaneous userspace
    applications may use the accelerator (although specific AFUs may
    support fewer). In this mode, the AFU sends a 16 bit context ID
    with each of its requests. This tells the PSL which context is
    associated with each operation. If the PSL can't translate an
    operation, the ID can also be accessed by the kernel so it can
    determine the userspace context associated with an operation.

MMIO space
==========

    A portion of the accelerator MMIO space can be directly mapped
    from the AFU to userspace. Either the whole space can be mapped or
    just a per context portion. The hardware is self describing, hence
    the kernel can determine the offset and size of the per context
    portion.

Interrupts
==========

    AFUs may generate interrupts that are destined for userspace. These
    are received by the kernel as hardware interrupts and passed onto
    userspace by a read syscall documented below.

    Data storage faults and error interrupts are handled by the kernel
    driver.

Work Element Descriptor (WED)
=============================

    The WED is a 64-bit parameter passed to the AFU when a context is
    started. Its format is up to the AFU hence the kernel has no
    knowledge of what it represents. Typically it will be the
    effective address of a work queue or status block where the AFU
    and userspace can share control and status information.

User API
========

1. AFU character devices
^^^^^^^^^^^^^^^^^^^^^^^^

    For AFUs operating in AFU directed mode, two character device
    files will be created. /dev/cxl/afu0.0m will correspond to a
    master context and /dev/cxl/afu0.0s will correspond to a slave
    context. Master contexts have access to the full MMIO space an
    AFU provides. Slave contexts have access to only the per process
    MMIO space an AFU provides.

    For AFUs operating in dedicated process mode, the driver will
    only create a single character device per AFU called
    /dev/cxl/afu0.0d. This will have access to the entire MMIO space
    that the AFU provides (like master contexts in AFU directed).

    The types described below are defined in include/uapi/misc/cxl.h

    The following file operations are supported on both slave and
    master devices.

    A userspace library libcxl is available here:

	https://github.com/ibm-capi/libcxl

    This provides a C interface to this kernel API.

open
----

    Opens the device and allocates a file descriptor to be used with
    the rest of the API.

    A dedicated mode AFU only has one context and only allows the
    device to be opened once.

    An AFU directed mode AFU can have many contexts, the device can be
    opened once for each context that is available.

    When all available contexts are allocated the open call will fail
    and return -ENOSPC.

    Note:
	  IRQs need to be allocated for each context, which may limit
          the number of contexts that can be created, and therefore
          how many times the device can be opened. The POWER8 CAPP
          supports 2040 IRQs and 3 are used by the kernel, so 2037 are
          left. If 1 IRQ is needed per context, then only 2037
          contexts can be allocated. If 4 IRQs are needed per context,
          then only 2037/4 = 509 contexts can be allocated.

ioctl
-----

    CXL_IOCTL_START_WORK:
        Starts the AFU context and associates it with the current
        process. Once this ioctl is successfully executed, all memory
        mapped into this process is accessible to this AFU context
        using the same effective addresses. No additional calls are
        required to map/unmap memory. The AFU memory context will be
        updated as userspace allocates and frees memory. This ioctl
        returns once the AFU context is started.

        Takes a pointer to a struct cxl_ioctl_start_work

            ::

                struct cxl_ioctl_start_work {
                        __u64 flags;
                        __u64 work_element_descriptor;
                        __u64 amr;
                        __s16 num_interrupts;
                        __s16 reserved1;
                        __s32 reserved2;
                        __u64 reserved3;
                        __u64 reserved4;
                        __u64 reserved5;
                        __u64 reserved6;
                };

            flags:
                Indicates which optional fields in the structure are
                valid.

            work_element_descriptor:
                The Work Element Descriptor (WED) is a 64-bit argument
                defined by the AFU. Typically this is an effective
                address pointing to an AFU specific structure
                describing what work to perform.

            amr:
                Authority Mask Register (AMR), same as the powerpc
                AMR. This field is only used by the kernel when the
                corresponding CXL_START_WORK_AMR value is specified in
                flags. If not specified the kernel will use a default
                value of 0.

            num_interrupts:
                Number of userspace interrupts to request. This field
                is only used by the kernel when the corresponding
                CXL_START_WORK_NUM_IRQS value is specified in flags.
                If not specified the minimum number required by the
                AFU will be allocated. The min and max number can be
                obtained from sysfs.

            reserved fields:
                For ABI padding and future extensions

    CXL_IOCTL_GET_PROCESS_ELEMENT:
        Get the current context id, also known as the process element.
        The value is returned from the kernel as a __u32.

mmap
----

    An AFU may have an MMIO space to facilitate communication with the
    AFU. If it does, the MMIO space can be accessed via mmap. The size
    and contents of this area are specific to the particular AFU. The
    size can be discovered via sysfs.

    In AFU directed mode, master contexts are allowed to map all of
    the MMIO space and slave contexts are allowed to only map the per
    process MMIO space associated with the context. In dedicated
    process mode the entire MMIO space can always be mapped.

    This mmap call must be done after the START_WORK ioctl.

    Care should be taken when accessing MMIO space. Only 32 and 64-bit
    accesses are supported by POWER8. Also, the AFU will be designed
    with a specific endianness, so all MMIO accesses should consider
    endianness (recommend endian(3) variants like: le64toh(),
    be64toh() etc). These endian issues equally apply to shared memory
    queues the WED may describe.

read
----

    Reads events from the AFU. Blocks if no events are pending
    (unless O_NONBLOCK is supplied). Returns -EIO in the case of an
    unrecoverable error or if the card is removed.

    read() will always return an integral number of events.

    The buffer passed to read() must be at least 4K bytes.

    The result of the read will be a buffer of one or more events,
    each event is of type struct cxl_event, of varying size::

            struct cxl_event {
                    struct cxl_event_header header;
                    union {
                            struct cxl_event_afu_interrupt irq;
                            struct cxl_event_data_storage fault;
                            struct cxl_event_afu_error afu_error;
                    };
            };

    The struct cxl_event_header is defined as

        ::

            struct cxl_event_header {
                    __u16 type;
                    __u16 size;
                    __u16 process_element;
                    __u16 reserved1;
            };

        type:
            This defines the type of event. The type determines how
            the rest of the event is structured. These types are
            described below and defined by enum cxl_event_type.

        size:
            This is the size of the event in bytes including the
            struct cxl_event_header. The start of the next event can
            be found at this offset from the start of the current
            event.

        process_element:
            Context ID of the event.

        reserved field:
            For future extensions and padding.

    If the event type is CXL_EVENT_AFU_INTERRUPT then the event
    structure is defined as

        ::

            struct cxl_event_afu_interrupt {
                    __u16 flags;
                    __u16 irq; /* Raised AFU interrupt number */
                    __u32 reserved1;
            };

        flags:
            These flags indicate which optional fields are present
            in this struct. Currently all fields are mandatory.

        irq:
            The IRQ number sent by the AFU.

        reserved field:
            For future extensions and padding.

    If the event type is CXL_EVENT_DATA_STORAGE then the event
    structure is defined as

        ::

            struct cxl_event_data_storage {
                    __u16 flags;
                    __u16 reserved1;
                    __u32 reserved2;
                    __u64 addr;
                    __u64 dsisr;
                    __u64 reserved3;
            };

        flags:
            These flags indicate which optional fields are present in
            this struct. Currently all fields are mandatory.

        address:
            The address that the AFU unsuccessfully attempted to
            access. Valid accesses will be handled transparently by the
            kernel but invalid accesses will generate this event.

        dsisr:
            This field gives information on the type of fault. It is a
            copy of the DSISR from the PSL hardware when the address
            fault occurred. The form of the DSISR is as defined in the
            CAIA.

        reserved fields:
            For future extensions

    If the event type is CXL_EVENT_AFU_ERROR then the event structure
    is defined as

        ::

            struct cxl_event_afu_error {
                    __u16 flags;
                    __u16 reserved1;
                    __u32 reserved2;
                    __u64 error;
            };

        flags:
            These flags indicate which optional fields are present in
            this struct. Currently all fields are Mandatory.

        error:
            Error status from the AFU. Defined by the AFU.

        reserved fields:
            For future extensions and padding

2. Card character device (powerVM guest only)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

    In a powerVM guest, an extra character device is created for the
    card. The device is only used to write (flash) a new image on the
    FPGA accelerator. Once the image is written and verified, the
    device tree is updated and the card is reset to reload the updated
    image.

open
----

    Opens the device and allocates a file descriptor to be used with
    the rest of the API. The device can only be opened once.

ioctl
-----

CXL_IOCTL_DOWNLOAD_IMAGE / CXL_IOCTL_VALIDATE_IMAGE:
    Starts and controls flashing a new FPGA image. Partial
    reconfiguration is not supported (yet), so the image must contain
    a copy of the PSL and AFU(s). Since an image can be quite large,
    the caller may have to iterate, splitting the image in smaller
    chunks.

    Takes a pointer to a struct cxl_adapter_image::

        struct cxl_adapter_image {
            __u64 flags;
            __u64 data;
            __u64 len_data;
            __u64 len_image;
            __u64 reserved1;
            __u64 reserved2;
            __u64 reserved3;
            __u64 reserved4;
        };

    flags:
        These flags indicate which optional fields are present in
        this struct. Currently all fields are mandatory.

    data:
        Pointer to a buffer with part of the image to write to the
        card.

    len_data:
        Size of the buffer pointed to by data.

    len_image:
        Full size of the image.

Sysfs Class
===========

    A cxl sysfs class is added under /sys/class/cxl to facilitate
    enumeration and tuning of the accelerators. Its layout is
    described in Documentation/ABI/testing/sysfs-class-cxl

Udev rules
==========

    The following udev rules could be used to create a symlink to the
    most logical chardev to use in any programming mode (afuX.Yd for
    dedicated, afuX.Ys for afu directed), since the API is virtually
    identical for each::

	SUBSYSTEM=="cxl", ATTRS{mode}=="dedicated_process", SYMLINK="cxl/%b"
	SUBSYSTEM=="cxl", ATTRS{mode}=="afu_directed", \
	                  KERNEL=="afu[0-9]*.[0-9]*s", SYMLINK="cxl/%b"