code:
Apart from Tahoe-LAFS (covered by [[todo/tahoe lfs for reals]] and [[forum/tips: special_remotes/hook with tahoe-lafs]]), [[special remotes]] (which I understand as real storage backends) for other other [peer network data stores](http://en.wikipedia.org/wiki/Distributed_data_store#Peer_network_node_data_stores_2) would be interesting.

I mean gnunet, freenet, BitTorrent (also trackerless).

Before dropping a file locally, the BitTorrent client should check that all parts are still available from the peers.

Of course, there is no guarantee assumed that the content won't disappear from the peer network in future: they act more like a cache rather than an archive on whose lifespan you decide. (I'm only not sure about gnunet now: whether there is a rule of dropping unused content from it, like in freenet.)

So, a copy in peer networks shouldn't be counted on by git-annex as much as a copy on a storage you control: probably, by efault, it shouldn't let you delete the local copy if there is a copy in a peer network unless you saved it somewhere else.

(Think of such a scenario: I could save some of my public large data on external disks/DVDs and keep them at home, and also put them onto peer networks with the same nterface of git-annex which I would be used to; I would also use the git-annex interface to check from time to time that the content is still present, i.e. "cached", on the peer networks. Whenever I'm away from home, and unexpectedly need to show this content to someone, or have a look at it for some reason, I could get it from the peer network "cache".)

Also networks like namecoin (derived from bitcoin) can be used as a key-value store. Despite being a peer network, a system like namecoin actually could offer the publisher more control over the lifespan of the content: he should be able to offer "financial" reward for others processing his key-value data. (But I'm not sure namecoin is designed reasonably for this reward system to work actually; but there might be appearing other similar systems.)

## A different view: extend the key-value backends with ways to look for the content in other content-addressable storage systems
We might want to look for the registered files in other [content-addressable storage systems](http://en.wikipedia.org/wiki/Content-addressable_storage#Open-source_implementations) (and also to be able to put the files there for storage).

For example:

* [**GNUnet**](http://en.wikipedia.org/wiki/Gnunet) uses its own hash format to address the content. git-annex could extend its own [[backends]] with a one to work with GNUnet, and by default have a built-in [[special remote|special remotes]] that would interact with GNUnet when looking for a content or storing some content. No special setup of the special remote in each repo should be necessary, because GNUnet is "global", so we'd just use the user's already configured GNUnet client. Just turning the builtin GNUnet special remote on or off should be an option (in the repo configuration, and when calling the commands that would query it, like `whereis`).
* **freenet** is similar.
* Similarly, a backend  for the hashes used in **BitTorrent** and **magnet links** could be used. If we want a trackerless mode, then probably it's a similar case for a "global"/built-in special remote that needs no local setup in each repo. Using a selected tracker would mean setting up a special remote in our repo.
* **Git**  itself can be viwed as place to look for the content. There could be a corresponding backend and a builtin special remote (needing no extra setup) to look for the content among the objects stored in the local Git repo. (What if we have a copy of a file that we've put under the control of git-annex in a previous Git commit? We could get it from the object store of Git.)
* **Venti**, [[**Tahoe-LAFS**|todo/tahoe lfs for reals]] would need a backend for their hashes, and a specially setup special remote in each repo where we'd like to use them--because these are not "global" system, we must setup the path to the instance of the filesystem we'd like to use.
* probably, there must be other interesting cases of this kind...
* (I'm also thinking about using somethng like a **bibliographic information** as a key, but then it wouldn't guarantee identical files: the same paper can be stored in different formats, etc. Cf. [**URNs**](http://en.wikipedia.org/wiki/Uniform_resource_name#Examples), via <http://it.slashdot.org/comments.pl?sid=3032489&cid=40907233>. Also, an URN like bibliographic information can't be computed from the file, it will have to be entered manually or obtained from another directory of URNs.)


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.14.141"
 subject="comment 1"
 date="2012-09-25T22:57:19Z"
 content="""
The best first step to adding such kinds of data stores to git-annex is probably to use the [[special_remotes/hook]] special remote to access them.
"""]]

repo_name: kakafarm/git

license: unknown


code:
[[!comment format=mdwn
 username="http://lj.rossia.org/users/imz/"
 ip="79.165.56.162"
 subject="comment 2"
 date="2012-09-25T23:29:49Z"
 content="""
I see. But then, as with Tahoe-LAFS, they also have their own formats for checksums, keys, which could be re-used in git-annex, and that needs special treatment.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.154"
 subject="comment 3"
 date="2014-03-18T19:49:09Z"
 content="""
The new [[special_remotes/external]] special remote's protocol has GETSTATE and SETSTATE commands that can be used to store per-remote values in the git-annex branch.

So, please go make these special remotes using it!
"""]]


code: Special remote support for [Ubuntu One](http://one.ubuntu.com) would be nice. They're [using propietary but open protocol](https://wiki.ubuntu.com/UbuntuOne/TechnicalDetails#ubuntuone-storageprotocol) based on [Google Protocol Buffers](http://code.google.com/p/protobuf/). There's [protobuf for Haskell](http://code.google.com/p/protobuf-haskell/) so it should be possible to compile [the protocol file](http://bazaar.launchpad.net/~ubuntuone-control-tower/ubuntuone-storage-protocol/trunk/view/head:/ubuntuone/storageprotocol/protocol.proto) to Haskell code and then use that to implement the native Ubuntu special remote.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.154"
 subject="comment 1"
 date="2014-03-18T20:02:14Z"
 content="""
I suggest that if someone wants to build this, they use the new external special remote protocol to do it.
"""]]


code:
[[!comment format=mdwn
 username="madduck"
 ip="2001:a60:f0fb:0:224:d7ff:fe04:c82c"
 subject="Ubuntu One to be discontinued"
 date="2014-04-07T05:11:52Z"
 content="""
Thanksfully, Canonical have stopped this silliness, Ubuntu One will be discontinued, so this todo can be marked \"wontfix\" and archived.
"""]]


code:
[swift](http://swift.openstack.org/) is the object storage of Openstack. Think S3, but fully open source. As it's backed by rackspace.com, NASA, Dell and several other major players, adoption rates will explode.

I can provide a test account soonish if need be, else rackspace.com if offering swift storage. Their API gateway lives at https://auth.api.rackspacecloud.com/v1.0

Richard


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 1"
 date="2011-05-14T10:04:36Z"
 content="""
I don't suppose this SWIFT api is compatible with the eucalytpus walrus api ?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 2"
 date="2011-05-14T15:00:51Z"
 content="""
It does offer a S3 compability layer, but that is de facto non-functioning as of right now.
"""]]


code:
[[!comment format=mdwn
 username="http://ertai.myopenid.com/"
 nickname="npouillard"
 subject="+1"
 date="2012-09-18T08:52:21Z"
 content="""
OVH (french IT company) is migrating its could storage infrastructure to swift/openstack and the next few weeks.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm5Z8wzidsumLYQnHdrVxpCx8vsd9llSlg"
 nickname="Emanuele"
 subject="comment 4"
 date="2013-11-22T15:52:42Z"
 content="""
OVH is offering free accounts with 25GB of storage on their hubiC service: https://hubic.com/en/

The API documentation (based on OAuth2 and OpenStack Swift) is available at https://api.hubic.com/
"""]]


code:
[[!comment format=mdwn
 username="http://schnouki.net/"
 nickname="Schnouki"
 subject="comment 5"
 date="2014-02-11T13:38:53Z"
 content="""
FYI, I just published a [special remote for hubiC](https://github.com/Schnouki/git-annex-remote-hubic) that uses the OAuth API for authentication and the SWIFT API for data transfers. Parts of it are specific for hubiC (mostly auth.py), but it should be possible to adapt it to other SWIFT providers as well.
"""]]


code:
As git annex keeps logs about file transfers anyway, it should be relatively easy to add traffic accounting to a repo. That would allow me to monitor how much traffic a given repo generates. As I might end up hosting git-annex repos for a few personal friends, I need/want a way to track the heavy hitters. -- RichiH

PS: If you ever plan to host git-annex similar branchable, this would probably be of interest to you, as well :)


code:
In regular repos, objects are stored in files of the form: .git/annex/objects/xY/z1/SHA1-.../SHA1-.... (scheme 1)

On (some) special remotes, the corresponding file is stored at: .../abc/def/SHA1-... (scheme 2)

I'm not sure why the same scheme as in .git/objects isn't used, but it would be useful that the two-directory prefix were the same for all objects stores.

My use case is: I synchronize a git repo, say containing photos, to a server on which I can't install git-annex. I want the server to store all annexed files. For the photos to be viewed online, the annex store must use the scheme 1 (because the symlinks point to files with scheme 1). So I need to rsync .git/annex/objects manually from my desktop, because a git-annex rsync remote uses scheme 2. On the other hand, the repo on this server is not known by git-annex (like it would if I used a rsync remote).

At least it would be valuable (to get around above problem) to have a plumbing command giving the 2-directory prefix from a given key, for example:

$ git annex prefix-dir SHA1-s2--3f786850e387550fdab836ed7e6dc881

7w/88

f18/122

Even if the 2 schemes were unified, this prefix-dir command would still be useful when hacking around git-annex (for now I need to maintain a dictionary structure).

Thanks a lot.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.243"
 subject="comment 1"
 date="2013-10-04T20:50:33Z"
 content="""
Using the mixed case hash directory names is not desirable because people want to use git-annex on a variety of filesystems and operating systems that treat them in a variety of broken ways. However, migrating to the all lower case hash directory names would require changing every git-annex symlink in every git repository, and I do not want to inflict that on my users.

It seems to me that the best solution to your problem is to install git-annex on your server, which should not be very hard.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnpdM9F8VbtQ_H5PaPMpGSxPe_d5L1eJ6w"
 nickname="Rafaël"
 subject="comment 2"
 date="2013-10-05T10:45:16Z"
 content="""
Thank you for prompt answer. I didn't know there were tarballs,
and indeed I managed to install them easily (although I had to
manually install glibc version 2.9, only 2.7 was installed).

I found that bare git repos also use lower case hash directory
names... I still would be happy with an optional migration to all
lower case, with a new key-value backend, to avoid this little
complication which happens sometimes (say when converting a repo
from non-bare to bare).
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnpdM9F8VbtQ_H5PaPMpGSxPe_d5L1eJ6w"
 nickname="Rafaël"
 subject="comment 3"
 date="2013-10-07T13:33:40Z"
 content="""
By the way, I just had the case above, i.e. convert a bare repo to non-bare. In order to keep the annex files, I cloned the bare one, and git-annex move'ed all annex content to the new repo, and to my surprise it was slow, as if all files where copied (or maybe they were only checksummed?), instead of being only renamed (old and new repos were on the same partition). So I restate that at least a command line tool giving the prefix dirs would be useful, to allow scripting for this kind of situation.
"""]]


code:
as far as I know, if you `git clone` locally a git-annex enabled repository, it will not have all the files available. you would need to use `git annex get` and all files would be copied over, wasting a significant amount of space.

`git-clone` has this `--local` flags which hardlinks objects in `.git/objects`, but also, maybe more interestingly, has a `--shared` option to simply tell git to look in another repo for objects. it seems to me git-annex could leverage those functionalities to avoid file duplication when using local repositories.

this would be especially useful for [ikiwiki](http://ikiwiki.info/forum/ikiwiki_and_big_files).

This is a [[wishlist]], but I would also welcome implementation pointers to do this myself, thanks! --[[anarcat]]

> [[dup|done]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.220"
 subject="comment 1"
 date="2013-09-25T17:14:28Z"
 content="""
git-annex uses cp --reflink=auto. So on a filesystem supporting COW file copies, like btrfs, `git annex get` will not use any disk space when getting from the same filesystem.

I do not like the idea of using hardlinks, because changing the file in one repository would change it in the other, which may not be desired.

[[union_mounting]] seems to cover this item pretty well, so I will close this as a duplicate.
"""]]


code: To implement things like my custom `git annex-push` without the dash, i.e. `git annex push`, an alias system for git-annex would be nice.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.154"
 subject="comment 1"
 date="2014-03-18T19:38:19Z"
 content="""
Why not just use git's alias system? It can't make `git annex $foo` aliases, but `git $foo` is shorter anyway..
"""]]


code: otherwise now it is impossible to judge how old those are when opening the dashboard


code:
[[!comment format=mdwn
 username="http://olivier.mehani.name/"
 nickname="olivier-mehani"
 subject="comment 1"
 date="2014-01-16T02:22:18Z"
 content="""
+1
"""]]


code:
When adding a Remote server through the webapp, it set-up a specific SSH key for later sync.

However, when the remote has been set-up manually, then later gets the assistant thrown at it, there doesn't appear to be a way to create and deploy such a key. This option could be offered in, e.g., the settings of the repo in the webapp.

> I feel this is out of scope for the assistant. If the user is able to
> manually add a git remote at the command line, then they should be able
> to configure ssh keys too. I don't want to complicate the assistant with
> a lot of code that tries to deal with half-configured things the user
> manually set up. [[wontfix|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="http://olivier.mehani.name/"
 nickname="olivier-mehani"
 subject="comment 1"
 date="2014-01-22T13:08:21Z"
 content="""
Hum, fair enough. The webapp might not be the best target. However, there might already be some logic to deploy the key, only not exposed in any UI (web or CLI).

However, I was under the impression that the key thath git-annex installs remotely is also limited to running git-annex-related tasks (using the command option; I cannot find any example in my configurations at the moment), rather than providing a generic login shell which happens to be used for git-annex.

The command to run on the remote server did not seem to be trivial (this is what I'm currently bumping against), and I guess there already are a few functions which create and install the authorized_files entry. Maybe providing, e.g., a

    git-annex installkey REMOTE

command, automating only this key-setup step for the user, would be good?
"""]]


code:
[[!comment format=mdwn
 username="http://olivier.mehani.name/"
 nickname="olivier-mehani"
 subject="Manual solution"
 date="2014-06-14T13:59:38Z"
 content="""
My problem stems from the fact that I manually git clone the git-annex repo, which prevents the assistant from creating the setup to use passwordless keys. I just reverse-engineered a working setup to work up what I was missing. I jot it down here for reference, but I guess the bottomline is that if you want to use the assistant with a repo, do it from the start.

I assume that the client has a clone of the git(-annex) repo of the server.

    client$ git clone server:annex

Our goal is to let git-annex on the client know that there is a specific key to use when connecting to server that will let it access the git-annex-shell (without a password). We first create the key.

    client:~$ ssh-keygen -t rsa -f ~/.ssh/git-annex/key.git-annex-server-user_annex
    [enter an empty passphrase]

We can then create a virtual SSH host on the client that will use this key to connect to the server, in client:~/.ssh/config:

    # Added manually for git-annex
    Host git-annex-server-user_annex
    	Hostname server
    	Port 22
    	IdentityFile ~/.ssh/git-annex/key.git-annex-server-user_annex
    	IdentitiesOnly yes
    	StrictHostKeyChecking yes

(git-annex seems to use .2F (%2F) to encode path separators in the filenames.)

The server then needs to know to let the key in, but only for git-annex in the specific folder. This is done in server:.ssh/authorized_keys:

    command=\"GIT_ANNEX_SHELL_DIRECTORY='annex' ~/.ssh/git-annex-shell\",no-agent-forwarding,no-port-forwarding,no-X11-forwarding,no-pty ssh-rsa AAAA... user@client

The bit starting with ssh-rsa is the public key created in client:.ssh/git-annex/key.git-annex-server-user_annex.pub at the same time as the private key.

Finally, all that remains is to change the remote in the client clone to use the virtual SSH host.

    client:~/annex $ git remote set-url origin ssh://user@git-annex-server-user_annex/~/annex
    client:~/annex $ git remote set-url origin --push ssh://user@git-annex-server-user_annex/~/annex

If everything worked, a sync from the client should now work without asking for a password, and starting the assistant will not either.

"""]]


code:
[[!comment format=mdwn
 username="http://olivier.mehani.name/"
 nickname="olivier-mehani"
 subject="comment 3"
 date="2014-06-14T14:15:55Z"
 content="""
After having done that on my first test repo, git-annex could sync, but failed to get the files.

    client:~/annex$ git annex get file
    get file (not available)
      Try making some of these repositories available:
      	12345678-90ab-cdef-1234567890abcdef1 -- user@server:~/annex [origin]

      (Note that these git remotes have annex-ignore set: origin)
    failed
    git-annex: get: 1 failed

The note helps: the problem is with the origin remote having annex-ignore set. git-annex therefore ignores it. This is easily fixed by just setting the flag to false.

    client:~/annex$ git config remote.origin.annex-ignore false

"""]]


code:
[[!comment format=mdwn
 username="http://olivier.mehani.name/"
 nickname="olivier-mehani"
 subject="comment 4"
 date="2014-07-09T01:09:33Z"
 content="""
And the ultimate, copy/pastable one, using shell variables:

    export GASERVER=server
    export GAUSER=user
    export GAPATH=/path

For the new client (using bash):

    export GASANEPATH=${GAPATH//\//.2F}
    export GASSHHOSTNAME=${GASERVER}-${GAUSER}_${GASANEPATH}
    ssh-keygen -t rsa -f  ~/.ssh/git-annex/key.git-annex-$GASSHHOSTNAME
    cat << EOF >> ~/.ssh/config
    # Added manually for git-annex
    Host git-annex-$GASSHHOSTNAME
      Hostname $GASERVER
      IdentityFile ~/.ssh/git-annex/key.git-annex-$GASSHHOSTNAME
      IdentitiesOnly yes
      StrictHostKeyChecking yes
    EOF
    ssh-copy-id -i ~/.ssh/git-annex/key.git-annex-$GASSHHOSTNAME $GAUSER@$GASERVER
    git remote add ${GASERVER/.*/} ssh://${GAUSER}@git-annex-${GASSHHOSTNAME}${GAPATH}
    git config remote.${GASERVER/.*/}.annex-ignore false

After the `ssh-copy-id` stage, the key can be used to get a full session. This
needs to be limited on the server, by prepending the following to the newly
added key in `.ssh/authorized_keys`, replacing `GAPATH` by the value of `$GAPATH`:

    command=\"GIT_ANNEX_SHELL_DIRECTORY='GAPATH' ~/.ssh/git-annex-shell\",no-agent-forwarding,no-port-forwarding,no-X11-forwarding,no-pty

From the client, one can make sure this has been limite properly by trying to log in with the key:

     ssh -i ~/.ssh/git-annex/key.git-annex-$GASSHHOSTNAME $GAUSER@$GASERVER -o IdentitiesOnly=yes

It should reply with the `git-annex-shell` helper complaing:

    git-annex-shell: bad parameters

    Usage: git-annex-shell [-c] command [parameters ...] [option ...]

    Plumbing commands:

    commit        DIRECTORY          commits any staged changes to the git-annex branch
    configlist    DIRECTORY          outputs relevant git configuration
    dropkey       DIRECTORY KEY ...  drops annexed content for specified keys
    gcryptsetup   DIRECTORY VALUE    sets up gcrypt repository
    inannex       DIRECTORY KEY ...  checks if keys are present in the annex
    notifychanges DIRECTORY          sends notification when git refs are changed
    recvkey       DIRECTORY KEY      runs rsync in server mode to receive content
    sendkey       DIRECTORY KEY      runs rsync in server mode to send content
    transferinfo  DIRECTORY KEY      updates sender on number of bytes of content received

... and this should be all set.
"""]]


code:
This is an idea to have a small placeholder file that is put into
place when the file's actual content is not available in the local
annex.

Details being discussed here: <http://bugs.debian.org/728552>


code:
git-annex can transfer data to or from any of a repository's git remotes.
Depending on where the remote is, the data transfer is done using rsync
(over ssh or locally), or plain cp (with copy-on-write
optimisations on supported filesystems), or using curl (for repositories
on the web). Some [[special_remotes]] are also supported that are not
traditional git remotes.

If a data transfer is interrupted, git-annex retains the partial transfer
to allow it to be automatically resumed later.

It's equally easy to transfer a single file to or from a repository,
or to launch a retrievel of a massive pile of files from whatever
repositories they are scattered amongst.

git-annex automatically uses whatever remotes are currently accessible,
preferring ones that are less expensive to talk to.

[[!img repomap.png caption="A real-world repository interconnection map
(generated by git-annex map)"]]


code:
Git-annex supports several levels of trust of a repository:

* semitrusted (default)
* untrusted
* trusted
* dead

## semitrusted

Normally, git-annex does not fully trust its stored [[location_tracking]]
information. When removing content, it will directly check
that other repositories have enough [[copies]].

Generally that explicit checking is a good idea. Consider that the current
[[location_tracking]] information for a remote may not yet have propagated
out. Or, a remote may have suffered a catastrophic loss of data, or itself
been lost.

There is still some trust involved here. A semitrusted repository is
depended on to retain a copy of the file content; possibly the only
[[copy|copies]].

(Being semitrusted is the default. The `git annex semitrust` command
restores a repository to this default, when it has been overridden.
The `--semitrust` option can temporarily restore a repository to this
default.)

## untrusted

An untrusted repository is not trusted to retain data at all. Git-annex
will retain sufficient [[copies]] of data elsewhere.

This is a good choice for eg, portable drives that could get lost. Or,
if a disk is known to be dying, you can set it to untrusted and let
`git annex fsck` warn about data that needs to be copied off it.

To configure a repository as untrusted, use the `git annex untrust`
command.

## trusted

Sometimes, you may have reasons to fully trust the location tracking
information for a repository. For example, it may be an offline
archival drive, from which you rarely or never remove content. Deciding
when it makes sense to trust the tracking info is up to you.

One way to handle this is just to use `--force` when a command cannot
access a remote you trust. Or to use `--trust` to specify a repisitory to
trust temporarily.

To configure a repository as fully and permanently trusted,
use the `git annex trust` command.

## dead

This is used to indicate that you have no trust that the repository
exists at all. It's appropriate to use when a drive has been lost,
or a directory irretrevably deleted. It will make git-annex avoid
even showing the repository as a place where data might still reside.


code:
Occasionally improvments are made to how git-annex stores its data,
that require an upgrade process to convert repositories made with an older
version to be used by a newer version. It's annoying, it should happen
rarely, but sometimes, it's worth it.

There's a committment that git-annex will always support upgrades from all
past versions. After all, you may have offline drives from an earlier
git-annex, and might want to use them with a newer git-annex.

git-annex will notice if it is run in a repository that
needs an upgrade, and refuse to do anything. To upgrade,
use the "git annex upgrade" command.

The upgrade process is guaranteed to be conflict-free. Unless you
already have git conflicts in your repository or between repositories.
Upgrading a repository with conflicts is not recommended; resolve the
conflicts first before upgrading git-annex.

## Upgrade events, so far

### v4 -> v5 (git-annex version 5.x)

The upgrade from v4 to v5 is handled
automatically, and only affects [[direct mode]] repositories.

This upgrade involves changing direct mode repositories to operate with
core.bare=true.

### v3 -> v4 (git-annex version 4.x)

v4 was only used for [[direct_mode]], to ensure that a version of git-annex
that understands direct mode was used with a direct mode repository.

### v2 -> v3 (git-annex version 3.x)

Involved moving the .git-annex/ directory into a separate git-annex branch.

After this upgrade, you should make sure you include the git-annex branch
when git pushing and pulling.

### tips for this upgrade

This upgrade is easier (and faster!) than the previous upgrades.
You don't need to upgrade every repository at once; it's sufficient
to upgrade each repository only when you next use it.

Example upgrade process:

	cd localrepo
	git pull
	git annex upgrade
	git commit -m "upgrade v2 to v3"
	git gc

### v1 -> v2 (git-annex version 0.20110316)

Involved adding hashing to .git/annex/ and changing the names of all keys.
Symlinks changed.

Also, hashing was added to location log files in .git-annex/.
And .gitattributes needed to have another line added to it.

Previously, files added to the SHA [[backends]] did not have their file
size tracked, while files added to the WORM backend did. Files added to
the SHA backends after the conversion will have their file size tracked,
and that information will be used by git-annex for disk free space checking.
To ensure that information is available for all your annexed files, see
[[upgrades/SHA_size]].

### tips for this upgrade

This upgrade can tend to take a while, if you have a lot of files.

Each clone of a repository should be individually upgraded.
Until a repository's remotes have been upgraded, git-annex
will refuse to communicate with them.

Start by upgrading one repository, and then you can commit
the changes git-annex staged during upgrade, and push them out to other
repositories. And then upgrade those other repositories. Doing it this
way avoids git-annex doing some duplicate work during the upgrade.

Example upgrade process:

	cd localrepo
	git pull
	git annex upgrade
	git commit -m "upgrade v1 to v2"
	git push

	ssh remote
	cd remoterepo
	git pull
	git annex upgrade
	...

### v0 -> v1 (git-annex version 0.04)

Involved a reorganisation of the layout of .git/annex/. Symlinks changed.

Handled more or less transparently, although git-annex was just 2 weeks
old at the time, and had few users other than Joey.

Before doing this upgrade, set annex.version:

	git config annex.version 0


code:
Before version 2 of the git-annex repository, files added to the SHA
[[backends]] did not have their file size tracked, while files added to the
WORM backend did. The file size information is used for disk free space
checking.

Files added to the SHA backends after the conversion will have their file
size tracked automatically. This disk free space checking is an optional
feature and since you're more likely to be using more recently added files,
you're unlikely to see any bad effect if you do nothing.

That said, if you have old files added to SHA backends that lack file size
tracking info, here's how you can add that info. After [[upgrading|upgrades]]
to repository version 2, in each repository run:

	git annex migrate
	git commit -m 'migrated keys for v2'

The usual caveats about [[tips/migrating_data_to_a_new_backend]]
apply; you will end up with unused keys that you can later clean up with
`git annex unused`.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkjvjLHW9Omza7x1VEzIFQ8Z5honhRB90I"
 nickname="Asheesh"
 subject="The fact that the keys changed causes merge conflicts"
 date="2012-06-25T00:28:59Z"
 content="""
FYI, I have run into a problem where if you 'git annex sync' between various 'git annex v3' repositories, if the different repositories are using different encodings of the SHA1 information (one including size, one not), then the 'git merge' will declare that they conflict.

There's no indication that 'git annex migrate' is the right tool to run, except from perusing the 'git annex' man page. In my opinion this is a major user interface problem.

-- Asheesh.
"""]]


code:
Unfortunately the initial gcrypt repository layout had to be changed
after git-annex version 4.20130920. If you have an encrypted git repository
created using version 4.20130920 or 4.20130909, you need to manually
upgrade it.

If you look at the contents of your gcrypt repository, you will
see a bare git repository, with a few three-letter subdirectories,
which are where git-annex stores its encrypted file contents:

<pre>
27f/  branches/  description  hooks/  objects/
HEAD  config     f37/         info/   refs/
</pre>

In the example above, the subdirectories are `27f` and `f37`.

All you need to do to transition is move those subdirectories
into an `annex/objects` directory.

	mkdir annex ; mkdir annex/objects ; mv 27f f37 annex/objects

Probably those are the only 3 letter things inside your git repository,
so this will probably work:

	mkdir annex ; mkdir annex/objects ; mv ??? annex/objects


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmj3kEGlCiy_Y-wb6AIBBnJ0B_SiPHV5Bo"
 nickname="Thomas"
 subject="small omission"
 date="2013-09-25T04:59:03Z"
 content="""
In your second example, `mv ??? annex` has to be `mv ??? annex/objects`, I think.
"""]]


code:
### use case: The Nomad

Alice is always on the move, often with her trusty netbook and a small
handheld terabyte USB drive, or a smaller USB keydrive. She has a server
out there on the net. She stores data, encrypted in the Cloud.

All these things can have different files on them, but Alice no longer
has to deal with the tedious process of keeping them manually in sync,
or remembering where she put a file. git-annex manages all these data
sources as if they were git remotes.
[[more about special remotes|special_remotes]]

When she has 1 bar on her cell, Alice queues up interesting files on her
server for later. At a coffee shop, she has git-annex download them to her
USB drive. High in the sky or in a remote cabin, she catches up on
podcasts, videos, and games, first letting git-annex copy them from
her USB drive to the netbook (this saves battery power).
[[more about transferring data|transferring_data]]

When she's done, she tells git-annex which to keep and which to remove.
They're all removed from her netbook to save space, and Alice knows
that next time she syncs up to the net, her changes will be synced back
to her server.
[[more about distributed version control|distributed_version_control]]


code:
### use case: The Archivist

Bob has many drives to archive his data, most of them
[[kept offline|tips/offline_archive_drives]], in a safe place.

With git-annex, Bob has a single directory tree that includes all
his files, even if their content is being stored offline. He can
reorganize his files using that tree, committing new versions to git,
without worry about accidentally deleting anything.

When Bob needs access to some files, git-annex can tell him which drive(s)
they're on, and easily make them available. Indeed, every drive knows what
is on every other drive.
[[more about location tracking|location_tracking]]

Bob thinks long-term, and so he appreciates that git-annex uses a simple
repository format. He knows his files will be accessible in the future
even if the world has forgotten about git-annex and git.
[[more about future-proofing|future_proofing]]

Run in a cron job, git-annex adds new files to archival drives at night. It
also helps Bob keep track of intentional, and unintentional copies of
files, and logs information he can use to decide when it's time to duplicate
the content of old drives.
[[more about backup copies|copies]]


code:
Users of this wiki, feel free to create a subpage of this one and talk
about yourself on it, within reason. You can link to it to sign your
comments.

List of users
=============
[[!inline pages="users/* and !users/*/* and !*/Discussion"
feeds=no archive=yes sort=title template=titlepage
rootpage="users" postformtext="Add yourself as an git-annex user:"]]


code:
I use git-annex to manage huge files, mostly video and audio attached to other git repositories (such as presentations), but I also use git-annex to manage my music collection across multiple devices. I also use it to manage the `ISO` images I download, podcasts, and youtube videos.

See <http://anarc.at/>.

[[!toc]]

My tips
=======

... or the ones I commented it, to be more precise.

[[!inline pages="tips/* and and link(users/anarcat)" sort=mtime feeds=no actions=yes archive=yes show=0]]

My todos
========

... same.

[[!inline pages="todo/* and !todo/done and !link(todo/done) and
link(users/anarcat)" sort=mtime feeds=no actions=yes archive=yes show=0]]

Done
----

[[!inline pages="todo/* and !todo/done and link(todo/done) and
link(users/anarcat)" feeds=no actions=yes archive=yes show=0]]

My bugs
=======

... same.

[[!inline pages="bugs/* and !bugs/done and !link(bugs/done) and
link(users/anarcat)" sort=mtime feeds=no actions=yes archive=yes show=0  template=buglist]]

Fixed
-----

[[!inline pages="bugs/* and !bugs/done and link(bugs/done) and
link(users/anarcat)" feeds=no actions=yes archive=yes show=0  template=buglist]]

Forum posts
===========

Forums where I posted.

[[!inline pages="forum/* and link(users/anarcat)" sort=mtime feeds=no actions=yes archive=yes show=0]]

Git annex dev news
==================

I find the recent changes a little too noisy, but i want more than just the [[news]], so i made up this little hybrid feed:

[[!inline pages="(news/* and !news/*/* and !*/Discussion) or (devblog/* and !devblog/*/*)" feedonly="yes"]]


code:
* **name**: chrysn
* **website**: <http://christian.amsuess.com/>
* **uses git-annex for** managing the family's photos (and possibly videos and music in the future).
* **likes git-annex because** it adds a layer of commit semantics over a regular file system without keeping everything in duplicate locally.
* **would like git-annex not to** be required any more at all when
  * git itself learns to use cow filesystems to avoid abundant disk usage, and
  * git gets better with shallow clones.

    git-annex might then still be a simpler tool that watches over what can be safely dropped from a particular shallow clone

    (the issues with shallow clones seem to relate primarily to shallow history; i haven't read anything about what would happen if all commits were checked out, but not all trees and blobs)


code:
Claes Wallin (韋嘉誠)

[[https://microca.st/clacke]]


code: [[!meta redir=clacke]]


code:
# François Marier

Free Software and Debian Developer. Lead developer of [Libravatar](https://www.libravatar.org)

* [Blog](http://feeding.cloud.geek.nz) and [homepage](http://fmarier.org)
* [Identica](http://identi.ca/fmarier) / [Twitter](https://twitter.com/fmarier)


code:
Greg Grossmeier

<http://grossmeier.net/about>


code:
Joey Hess <joey@kitenet.net>
<http://kitenet.net/~joey/>


code:
Extremely grateful user of git-annex.

[[http://sameer.sbuddhe.net/]]


code:
Tobias Ussing

See:

* [[tips/flickrannex]] - [[https://github.com/TobiasTheViking/flickrannex/]]
* [[tips/imapannex]] - [[https://github.com/TobiasTheViking/imapannex]]
* [[tips/dropboxannex]] - [[https://github.com/TobiasTheViking/dropboxannex]]
* [[tips/skydriveannex]] - [[https://github.com/TobiasTheViking/skydriveannex]]
* [[tips/googledriveannex]] - [[https://github.com/TobiasTheViking/googledriveannex]]
* [[tips/owncloudannex]] - [[https://github.com/TobiasTheViking/owncloudannex]]
* [[tips/megaannex]] - [[https://github.com/TobiasTheViking/megaannex]]
* [[forum/nntp__47__usenet_special_remote/]]


code:
Talks and screencasts about git-annex.

These videos are also available in a public git-annex repository
`git clone https://downloads.kitenet.net/.git/`

[[!inline pages="./videos/* and !./videos/*/* and !*/Discussion" show="2"]]

[[!inline pages="./videos/* and !./videos/*/* and !*/Discussion" show="0" archive=yes skip=2 feeds=no]]


code:
<video controls
src="http://video.fosdem.org/2012/lightningtalks/git_annex___manage_files_with_git,_without_checking_their_contents_into_git.webm"></video>

A 15 minute introduction to git-annex,
presented by Richard Hartmann at FOSDEM 2012.

[[!meta date="1 Jan 2012"]]
[[!meta title="git-annex presentation by Richard Hartmann at FOSDEM 2012"]]


code:
<video controls>
<source type="video/mp4" src="http://mirror.linux.org.au/linux.conf.au/2013/mp4/gitannex.mp4">
<source type="video/ogg" src="http://mirror.linux.org.au/linux.conf.au/2013/ogv/gitannex.ogv">
</video>

A 45 minute talk and demo of git-annex and the assistant), presented by Joey Hess at LCA 2013.

[[!meta date="1 Feb 2013"]]
[[!meta title="git-annex presentation by Joey Hess at Linux.Conf.Au 2013"]]


code:
<video controls width=400>
<source src="https://downloads.kitenet.net/videos/git-annex/git-annex-assistant-archiving.ogv">
</video>

A 9 minute screencast
covering archiving your files with the [[git-annex assistant|/assistant]].


code:
<video controls width=400>
<source src="https://downloads.kitenet.net/videos/git-annex/git-annex-assistant-intro.ogv">
</video>

A 8 minute screencast
introducing the [[git-annex assistant|/assistant]].


code:
[[!comment format=mdwn
 username="modules"
 ip="85.16.227.39"
 subject="Great screencast"
 date="2013-03-16T14:16:37Z"
 content="""
I am starting to understand the concept :) Thank you.

The assistant is working perfect with local repos and removable drives but i have problems to setup a remote repos. On a fresh debian server with base setup and rsync installed and ssh-keys for login. I got a green \"Scanned 93.xxx.xx.xxx_annex\" message on dashboard, but there is never a finished transfer. Its \"queued\" on all files and nothing seems to happens on clicking the \"play\" button behind files (x-ing files removes them from dashboard). Also i saw there is a second option in your screencast for remote servers (\"Use a git reposoitory on the server\") which does not show up on my side. Do i need to setup git-annex on server for second option? (with git-annex version 4.20130314).

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 2"
 date="2013-03-16T16:04:30Z"
 content="""
@modules: Yes, you're only given the option to use a git repository on the server if it has both git and git-annex installed. You can install any version of git-annex there. For example, Debian stable ships with one that will work.
I plan to make that screen clearer when the git repository option is not available.

I'm not sure what'd going on with your rsync transfers not running. I can say that if the \"play\" icon is visible, the transfer has been paused. While a transfer is running, the \"pause\" icon is visible instead, to let you pause it.
This may be as simple as you having misunderstood the icons and paused the currently running transfer, which prevents any transfers from running. If not, suggest you enable debug logs in the Preferences page, and consider filing a [[bug_report|bugs]].
"""]]


code:
<video controls width=400>
<source src="https://downloads.kitenet.net/videos/git-annex/git-annex-lan.webm">
</video>

A 10 minute screencast
showing how to get started using the [[git-annex assistant|/assistant]],
including sharing files on a local network, and installation on a server.


code:
[[!comment format=mdwn
 username="http://alan.petitepomme.net/"
 nickname="Alan Schmitt"
 subject="How is the binary found on the server?"
 date="2014-06-05T12:05:40Z"
 content="""
Great screencast! There is one thing that intrigues me: since you simply uncompress the binary in some folder, how does gitannex figures out where the binary lives there?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="magic"
 date="2014-06-10T16:13:35Z"
 content="""
In the screencast, I run git-annex after unpacking it, and this lets it register where it's installed at.

So it is important to run git-annex after you unpack it.
"""]]


code:
[[!comment format=mdwn
 username="http://alan.petitepomme.net/"
 nickname="Alan Schmitt"
 subject="Where is it registered?"
 date="2014-06-11T11:27:02Z"
 content="""
I'm curious: how do you register it? Does it only work on Linux? (I've had to add paths to my setup under OS X, so it would be great if there was such a simpler way.)
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 4"
 date="2014-06-11T19:56:49Z"
 content="""
Running git-annex will also register it on OSX. The registration just consists of making a ~/.ssh/git-annex-shell that runs the real git-annex-shell. The assistant detects when it needs to use that wrapper when setting up a repository.
"""]]


code:
<video controls width=400>
<source src="https://downloads.kitenet.net/videos/git-annex/git-annex-xmpp-pairing.ogv">
</video>

A 6 minute screencast
showing how to share files between your computers in different locations,
such as home and work.


code:
A screencast demoing the git-annex assistant syncing between Nicaragua
and the United Kingdom for the first time.

<video controls src="http://joeyh.name/screencasts/git-annex-assistant.ogg"></video>

[video](http://joeyh.name/screencasts/git-annex-assistant.ogg)

[[!meta date="Thu Jul 5 16:36:06 2012 -0600"]]


code:
A quick screencast demoing an experimental new feature,
[[tips/metadata_driven_views]].

<video controls src="https://downloads.kitenet.net/videos/git-annex/git-annex_views_demo.ogg"></video>

[video](https://downloads.kitenet.net/videos/git-annex/git-annex_views_demo.ogg)

Credits:

* RichiH for <https://github.com/RichiH/conference_proceedings>
* Michi for keyboard cat cameo


code:
A quick screencast demoing the `git annex watch` daemon.

<video controls src="http://joeyh.name/screencasts/git-annex-watch.ogg"></video>

[video](http://joeyh.name/screencasts/git-annex-watch.ogg)

[[!meta date="Mon Jun 11 16:02:14 2012 -0400"]]


code:
A quick screencast demoing the early `git annex webapp` and
automatic USB drive mount detection and syncing.

<video controls src="http://joeyh.name/screencasts/git-annex-webapp.ogg"></video>

[video](http://joeyh.name/screencasts/git-annex-webapp.ogg)

[[!meta date="Sun Jul 29 14:41:41 2012 -0400"]]


code:
A walkthrough of the basic features of git-annex.

[[!toc]]

[[!inline feeds=no trail=yes show=0 template=walkthrough pagenames="""
	walkthrough/creating_a_repository
	walkthrough/adding_a_remote
	walkthrough/adding_files
	walkthrough/renaming_files
	walkthrough/getting_file_content
	walkthrough/syncing
	walkthrough/transferring_files:_When_things_go_wrong
	walkthrough/removing_files
	walkthrough/removing_files:_When_things_go_wrong
	walkthrough/modifying_annexed_files
	walkthrough/using_ssh_remotes
	walkthrough/moving_file_content_between_repositories
	walkthrough/quiet_please:_When_git-annex_seems_to_skip_files
	walkthrough/using_tags_and_branches
	walkthrough/unused_data
	walkthrough/fsck:_verifying_your_data
	walkthrough/fsck:_when_things_go_wrong
	walkthrough/backups
	walkthrough/automatically_managing_content
	walkthrough/more
"""]]


code:
Like any other git repository, git-annex repositories have remotes.
Let's start by adding a USB drive as a remote.

	# sudo mount /media/usb
	# cd /media/usb
	# git clone ~/annex
	# cd annex
	# git annex init "portable USB drive"
	# git remote add laptop ~/annex
	# cd ~/annex
	# git remote add usbdrive /media/usb/annex

This is all standard ad-hoc distributed git repository setup.
The only git-annex specific part is telling it the name
of the new repository created on the USB drive.

Notice that both repos are set up as remotes of one another. This lets
either get annexed files from the other. You'll want to do that even
if you are using git in a more centralized fashion.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 1"
 date="2011-03-19T01:18:49Z"
 content="""
After doing the above with two required copy per file, `git annex fsck` complained that I had only one copy per file even though I had created my clone, already. Once I `git pull`ed from the second repo, not getting any changes for obvious reasons, `git annex fsck` was happy. So I am not sure how my addition was incorrect. -- RichiH
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 2"
 date="2011-03-19T15:35:38Z"
 content="""
Yes, you have to pull down location tracking information in order for fsck to be satisfied in that situation. But since this is a walkthrough, and neither fsck or numcopies settings are mentioned until later, it's ok for this pull to be described a few steps along in [[getting file content]].

"""]]


code:
[[!comment format=mdwn
 username="http://dieter-be.myopenid.com/"
 nickname="dieter"
 subject="comment 3"
 date="2011-04-02T20:24:33Z"
 content="""
 * why the `git remote add laptop ~/annex` ? this remote already exists under the name origin.
 * doesn't the last command need to be `git remote add usbdrive /media/usb/annex`? because the actual repo would be in /media/usb/annex, not /media/usb?
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 4"
 date="2011-04-03T02:32:17Z"
 content="""
Good spotting on the last line, fixed.

The laptop remote is indeed redundant, but it leads to clearer views of what is going on later in the walkthrough (\"git pull laptop master\", \"(copying from laptop...)\"). And if the original clone is made from a central bare repo, this reinforces that you'll want to set up remotes for other repos on the computer.
"""]]


code:
	# cd ~/annex
	# cp /tmp/big_file .
	# cp /tmp/debian.iso .
	# git annex add .
	add big_file (checksum...) ok
	add debian.iso (checksum...) ok
	# git commit -a -m added

When you add a file to the annex and commit it, only a symlink to
the content is committed to git. The content itself is stored in
git-annex's backend, `.git/annex/` (or in [[direct_mode]] the file
is left as-is).


code:
Once you have multiple repositories, and have perhaps configured numcopies,
any given file can have many more copies than is needed, or perhaps fewer
than you would like. How to manage this?

The whereis subcommand can be used to see how many copies of a file are known,
but then you have to decide what to get or drop. In this example, there
are perhaps not enough copies of the first file, and too many of the second
file.

	# cd /media/usbdrive
	# git annex whereis
	whereis my_cool_big_file (1 copy)
		0c443de8-e644-11df-acbf-f7cd7ca6210d  -- laptop
	whereis other_file (3 copies)
		0c443de8-e644-11df-acbf-f7cd7ca6210d  -- laptop
		62b39bbe-4149-11e0-af01-bb89245a1e61  -- usb drive [here]
		7570b02e-15e9-11e0-adf0-9f3f94cb2eaa  -- backup drive

What would be handy is some automated versions of get and drop, that only
gets a file if there are not yet enough copies of it, or only drops a file
if there are too many copies. Well, these exist, just use the --auto option.

	# git annex get --auto --numcopies=2
	get my_cool_big_file (from laptop...) ok
	# git annex drop --auto --numcopies=2
	drop other_file ok

With two quick commands, git-annex was able to decide for you how to
work toward having two copies of your files.

	# git annex whereis
	whereis my_cool_big_file (2 copies)
		0c443de8-e644-11df-acbf-f7cd7ca6210d  -- laptop
		62b39bbe-4149-11e0-af01-bb89245a1e61  -- usb drive [here]
	whereis other_file (2 copies)
		0c443de8-e644-11df-acbf-f7cd7ca6210d  -- laptop
		7570b02e-15e9-11e0-adf0-9f3f94cb2eaa  -- backup drive

The --auto option can also be used with the copy command,
again this lets git-annex decide whether to actually copy content.

The above shows how to use --auto to manage content based on the number
of copies. It's also possible to configure, on a per-repository basis,
which content is desired. Then --auto also takes that into account
see [[preferred_content]] for details.


code:
git-annex can be configured to require more than one copy of a file exists,
as a simple backup for your data. This is controlled by the
numcopies setting, which defaults to 1 copy. Let's
change that to require 2 copies, and send a copy of every file
to a USB drive.

	# git annex numcopies 2
	# git annex copy . --to usbdrive

Now when we try to `git annex drop` a file, it will verify that it
knows of 2 other repositories that have a copy before removing its
content from the current repository.

The numcopies setting used above is the global default.
You can also vary the number of copies needed, depending on the file name.
So, if you want 3 copies of all your flac files, but only 1 copy of oggs:

	# echo "*.ogg annex.numcopies=1" >> .gitattributes
	# echo "*.flac annex.numcopies=3" >> .gitattributes

Or, you might want to make a directory for important stuff, and configure
it so anything put in there is backed up more thoroughly:

	# mkdir important_stuff
	# echo "* annex.numcopies=3" > important_stuff/.gitattributes

For more details about the numcopies setting, see [[copies]].


code:
[[!comment format=mdwn
 username="madduck"
 ip="2001:a60:f0fb:0:224:d7ff:fe04:c82c"
 subject="Warn while inconsistent"
 date="2014-04-06T20:44:17Z"
 content="""
Sure, git-annex prevents me from dropping files unless there are numcopies around elsewhere, but shouldn't it also ensure that numcopies cannot be set unless that requirement is already met?

Furthermore, shouldn't it ensure that when new files are added, they are automatically distributed to fulfill the requirement?
"""]]


code:
This is very straightforward. Just tell it a description of the repository.

	# mkdir ~/annex
	# cd ~/annex
	# git init
	# git annex init "my laptop"


code:
You can use the fsck subcommand to check for problems in your data. What
can be checked depends on the key-value [[backend|backends]] you've used
for the data. For example, when you use the SHA1 backend, fsck will verify
that the checksums of your files are good. Fsck also checks that the
[[numcopies|copies]] setting is satisfied for all files.

	# git annex fsck
	fsck some_file (checksum...) ok
	fsck my_cool_big_file (checksum...) ok
	...

You can also specify the files to check.  This is particularly useful if
you're using sha1 and don't want to spend a long time checksumming everything.

	# git annex fsck my_cool_big_file
	fsck my_cool_big_file (checksum...) ok


code:
Fsck never deletes possibly bad data; instead it will be moved to
`.git/annex/bad/` for you to recover. Here is a sample of what fsck
might say about a badly messed up annex:

	# git annex fsck
	fsck my_cool_big_file (checksum...)
	git-annex: Bad file content; moved to .git/annex/bad/SHA1:7da006579dd64330eb2456001fd01948430572f2
	git-annex: ** No known copies exist of my_cool_big_file
	failed
	fsck important_file
	git-annex: Only 1 of 2 copies exist. Run git annex get somewhere else to back it up.
	failed
	git-annex: 2 failed


code:
A repository does not always have all annexed file contents available.
When you need the content of a file, you can use "git annex get" to
make it available.

We can use this to copy everything in the laptop's annex to the
USB drive.

	# cd /media/usb/annex
	# git annex sync laptop
	# git annex get .
	get my_cool_big_file (from laptop...) ok
	get iso/debian.iso (from laptop...) ok


code:
Normally, the content of files in the annex is prevented from being modified.
(Unless your repository is using [[direct_mode]].)

That's a good thing, because it might be the only copy, you wouldn't
want to lose it in a fumblefingered mistake.

	# echo oops > my_cool_big_file
	bash: my_cool_big_file: Permission denied

In order to modify a file, it should first be unlocked.

	# git annex unlock my_cool_big_file
	unlock my_cool_big_file (copying...) ok

That replaces the symlink that normally points at its content with a copy
of the content. You can then modify the file like any regular file. Because
it is a regular file.

(If you decide you don't need to modify the file after all, or want to discard
modifications, just use `git annex lock`.)

When you `git commit`, git-annex's pre-commit hook will automatically
notice that you are committing an unlocked file, and add its new content
to the annex. The file will be replaced with a symlink to the new content,
and this symlink is what gets committed to git in the end.

	# echo "now smaller, but even cooler" > my_cool_big_file
	# git commit my_cool_big_file -m "changed an annexed file"
	add my_cool_big_file ok
	[master 64cda67] changed an annexed file
	 1 files changed, 1 insertions(+), 1 deletions(-)

There is one problem with using `git commit` like this: Git wants to first
stage the entire contents of the file in its index. That can be slow for
big files (sorta why git-annex exists in the first place). So, the
automatic handling on commit is a nice safety feature, since it prevents
the file content being accidentally committed into git. But when working with
big files, it's faster to explicitly add them to the annex yourself
before committing.

	# echo "now smaller, but even cooler yet" > my_cool_big_file
	# git annex add my_cool_big_file
	add my_cool_big_file ok
	# git commit my_cool_big_file -m "changed an annexed file"


code:
[[!comment format=mdwn
 username="http://lj.rossia.org/users/imz/"
 ip="79.165.56.162"
 subject="sorta why git-annex exists in the first place -- not only the slow index "
 date="2012-09-25T22:04:01Z"
 content="""
> Git wants to first stage the entire contents of the file in its index. That can be slow for big files (sorta why git-annex exists in the first place)

I think that git-annex's usefulness is not only because of the Git's index overhead: I like its idea because it will help track the copies in the \"[[special remotes]]\", which are not Git because

* they are either not under my control (e.g., web URLs),
* or because it's not convenient to hold a Git repo there (an external disk/DVD with files can be viewed easily by a human, but imposing a Git repo structure on it would either at least double the consume space: for the history of the commits and for the working dir, or make it \"unreadable\" for a human, if it is a bare repo);
* or because it's nearly impossible to put a Git repo on a storage like peer networks without special tools.
"""]]


code:
[[!comment format=mdwn
 username="http://yarikoptic.myopenid.com/"
 nickname="site-myopenid"
 subject="feature request: unlock --drop"
 date="2013-05-30T20:26:52Z"
 content="""
ATM unlock copies original file for modifications, so that original copy remains under annex and possibly to-be-edited copy created.
But if I am \"unlock\"ing file I might simply not be interested in a previous copy and want to maintain only a single (possibly edited) new copy.
What if there was a mode where the actual file is simply moved into \"unlocked\" location for editing, thus effectively dropping the actual content from git annex.  That would allow to not inquire lengthy copying/wasting local space.  If then I would need a previous copy I would just \"get\" it again.
"""]]


code:
So ends the walkthrough. By now you should be able to use git-annex.

Want more? See [[tips]] for lots more features and advice.


code:
Often you will want to move some file contents from a repository to some
other one. For example, your laptop's disk is getting full; time to move
some files to an external disk before moving another file from a file
server to your laptop. Doing that by hand (by using `git annex get` and
`git annex drop`) is possible, but a bit of a pain. `git annex move`
makes it very easy.

	# git annex move my_cool_big_file --to usbdrive
	move my_cool_big_file (to usbdrive...) ok
	# git annex move video/hackity_hack_and_kaxxt.mov --from fileserver
	move video/hackity_hack_and_kaxxt.mov (from fileserver...)
	SHA256-s86050597--6ae2688bc533437766a48aa19f2c06be14d1bab9c70b468af445d4f07b65f41e   100%   82MB 199.1KB/s   07:02
	ok


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 1"
 date="2011-03-22T23:41:51Z"
 content="""
I may be missing something obvious, but when I copy to a remote repository, the object files are created, but no softlinks are created. When I pull everything from the remote, it pulls only files the local repo knows about already.

        A
       / \
      B   C

Moving from B to A creates no symlinks in A but the object files are moved to A. Copying back from A to B restores the object files in B and keeps them in A.

Copying from A to an empty C does not create any object files nor symlinks. Copying from C to A creates no symlinks in A but the object files are copied to A.

-- RichiH

"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 2"
 date="2011-03-23T00:38:10Z"
 content="""
`git annex move` only moves content. All symlink management is handled by git, so you have to keep repositories in sync using git as you would any other repo. When you `git pull B` in A, it will get whatever symlinks were added to B.

(It can be useful to use a central bare repo and avoid needing to git pull from one repo to another, then you can just always push commits to the central repo, and pull down all changes from other repos.)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 3"
 date="2011-03-23T02:07:49Z"
 content="""
Ah yes, I feel kinda stupid in hindsight.

As the central server is most likely a common use case, would you object if I added that to the walkthrough? If you have any best practices on how to automate a push with every copy to a bare remote? AFAIK, git does not store information about bare/non-bare remotes, but this could easily be put into .git/config by git annex.

-- RichiH
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 4"
 date="2011-03-23T15:28:00Z"
 content="""
I would not mind if the walkthrough documented the central git repo case. But I don't want to complicate it unduely (it's long enough), and it's important that the fully distributed case be shown to work, and I assume that people already have basic git knowledge, so documenting the details of set up of a bare git repo is sorta out of scope. (There are also a lot of way to do it, using github, or gitosis, or raw git, etc.)
"""]]


code:
One behavior of git-annex is sometimes confusing at first, but it turns out
to be useful once you get to know it.

	# git annex drop *
	#

Why didn't git-annex seem to do anything despite being asked to drop all the
files? Because it checked them all, and none of them are present.

Most git-annex commands will behave this way when they're able to quickly
check that nothing needs to be done about a file.

Running a git-annex command without specifying any file name will
make git-annex look for files in the current directory and its
subdirectories. So, we can add all new files to the annex easily:

	# echo hi > subdir/subsubdir/newfile
	# git annex add
	add subdir/subsubdir/newfile ok

When doing this kind of thing, having nothing shown for files
that it doesn't need to act on is useful because it prevents swamping
you with output. You only see the files it finds it does need to act on.

So remember: If git-annex seems to not do anything when you tell it to, it's
not being lazy -- It's checked that nothing needs to be done to get to the
state you asked for!


code:
When you're using git-annex you can `git rm` a file just like you usually
would with git. Just like with git, this removes the file from your work
tree, but it does not remove the file's content from the git repository.
If you check the file back out, or revert the removal, you can get it back.

Git-annex adds the ability to remove the content of a file from your local
repository to save space. This is called "dropping" the file.

You can always drop files safely. Git-annex checks that some other
repository still has the file before removing it.

	# git annex drop iso/debian.iso
	drop iso/Debian_5.0.iso ok

Once dropped, the file will still appear in your work tree as a broken symlink.
You can use `git annex get` to as usual to get this file back to your local
repository.


code:
[[!comment format=mdwn
 username="DavidEdmondson"
 subject="Is it necessary to commit after the 'drop'?"
 date="2011-09-05T15:43:25Z"
 content="""
In fact is it possible? Nothing changed as far as git is concerned.

"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 2"
 date="2011-09-05T15:59:27Z"
 content="""
Good catch. It used to be necessary before there was a git-annex branch, but not now.
"""]]


code:
Before dropping a file, git-annex wants to be able to look at other
remotes, and verify that they still have a file. After all, it could
have been dropped from them too. If the remotes are not mounted/available,
you'll see something like this.

	# git annex drop important_file other.iso
	drop important_file (unsafe)
	  Could only verify the existence of 0 out of 1 necessary copies
	  Unable to access these remotes: usbdrive
	  Try making some of these repositories available:
	   	58d84e8a-d9ae-11df-a1aa-ab9aa8c00826  -- portable USB drive
	   	ca20064c-dbb5-11df-b2fe-002170d25c55  -- backup SATA drive
	  (Use --force to override this check, or adjust numcopies.)
	failed
	drop other.iso (unsafe)
	  Could only verify the existence of 0 out of 1 necessary copies
          No other repository is known to contain the file.
	  (Use --force to override this check, or adjust numcopies.)
	failed

Here you might --force it to drop `important_file` if you [[trust]] your backup.
But `other.iso` looks to have never been copied to anywhere else, so if
it's something you want to hold onto, you'd need to transfer it to
some other repository before dropping it.


code:
	# cd ~/annex
	# git mv big_file my_cool_big_file
	# mkdir iso
	# git mv debian.iso iso/
	# git commit -m moved

You can use any normal git operations to move files around, or even
make copies or delete them.

Notice that, since annexed files are represented by symlinks,
the symlink will break when the file is moved into a subdirectory.
But, git-annex will fix this up for you when you commit --
it has a pre-commit hook that watches for and corrects broken symlinks.

## Direct mode

Note that these git commands only work when git-annex is using indirect mode. Repositories created by the [[assistant]] are in [[direct_mode]]. Running 'git mv' in direct mode will give you an error:

    fatal: This operation must be run in a work tree


code:
Notice that in the [[previous example|getting_file_content]], `git annex
sync` was used. This lets git-annex know what has changed in the other
repositories like the laptop, and so it knows about the files present there and can
get them.

Let's look at what the sync command does in more detail:

	# cd /media/usb/annex
	# git annex sync
	commit
	nothing to commit (working directory clean)
	ok
	pull laptop
	ok
	push laptop
	ok

After you run sync, the git repository will be updated with all changes
made to its remotes, and any changes in the git repository will be pushed
out to its remotes, where a sync will get them. This is especially useful
when using git in a distributed fashion, without a [[central bare
repository|tips/centralized_git_repository_tutorial]]. See [[sync]] for
details.

By default `git annex sync` only syncs the metadata about your
files that is stored in git. It does not sync the contents of files, that
are managed by git-annex. To do that, you can use `git annex sync --content`


code:
After a while, you'll have several annexes, with different file contents.
You don't have to try to keep all that straight; git-annex does
[[location_tracking]] for you. If you ask it to get a file and the drive
or file server is not accessible, it will let you know what it needs to get
it:

	# git annex get video/hackity_hack_and_kaxxt.mov
	get video/_why_hackity_hack_and_kaxxt.mov (not available)
	  Unable to access these remotes: usbdrive, server
	  Try making some of these repositories available:
	  	5863d8c0-d9a9-11df-adb2-af51e6559a49  -- my home file server
	   	58d84e8a-d9ae-11df-a1aa-ab9aa8c00826  -- portable USB drive
	   	ca20064c-dbb5-11df-b2fe-002170d25c55  -- backup SATA drive
	failed
	# sudo mount /media/usb
	# git annex get video/hackity_hack_and_kaxxt.mov
	get video/hackity_hack_and_kaxxt.mov (from usbdrive...) ok


code:
It's possible for data to accumulate in the annex that no files in any
branch point to anymore. One way it can happen is if you `git rm` a file
without first calling `git annex drop`. And, when you modify an annexed
file, the old content of the file remains in the annex. Another way is when
migrating between key-value [[backends]].

This might be historical data you want to preserve, so git-annex defaults to
preserving it. So from time to time, you may want to check for such data:

	# git annex unused
	unused . (checking for unused data...)
	  Some annexed data is no longer used by any files in the repository.
	    NUMBER  KEY
	    1       SHA256-s86050597--6ae2688bc533437766a48aa19f2c06be14d1bab9c70b468af445d4f07b65f41e
	    2       SHA1-s14--f1358ec1873d57350e3dc62054dc232bc93c2bd1
	  (To see where data was previously used, try: git log --stat -S'KEY')
	  (To remove unwanted data: git-annex dropunused NUMBER)
	ok

After running `git annex unused`, you can follow the instructions to examine
the history of files that used the data, and if you decide you don't need that
data anymore, you can easily remove it from your local repository.

	# git annex dropunused 1
	dropunused 1 ok

Hint: To drop a lot of unused data, use a command like this:

	# git annex dropunused 1-1000

Rather than removing the data, you can instead send it to other
repositories:

	# git annex copy --unused --to backup
	# git annex move --unused --to archive


code:
[[!comment format=mdwn
 username="bremner"
 ip="156.34.89.108"
 subject="finding data that isn't unused, but should be."
 date="2012-10-17T20:32:11Z"
 content="""
Sometimes links to annexed data still exists on some branch, when it was supposed to be dropped. Here is how I found these; perhaps there is a simpler way.

    % git annex find --format '${key}\n' | sort > /tmp/known-keys
    % find .git/annex/objects -type f -exec basename {} \; | sort  > /tmp/local-keys
    % comm -23 /tmp/local-keys /tmp/known-keys

to look for what branch these are on, try

    % git log --stat --all -S$key

for one of the keys output above. In my case it was the same remote branch keeping them all alive.

*EDIT* sort key lists to make comm work properly

"""]]


code:
Another [[special_remote|special_remotes]] that git-annex can use is
a [[special_remotes/bup]] repository. Bup stores large file contents
in a git repository of its own, with deduplication. Combined with
git-annex, you can have git on both the frontend and the backend.

Here's how to create a bup remote, and describe it.

[[!template id=note text="""
Instead of specifying a remote system, you could choose to make a bup
remote that is only accessible on the current system, by passing
"buprepo=/big/mybup".
"""]]

	# git annex initremote mybup type=bup encryption=none buprepo=example.com:/big/mybup
	initremote bup (bup init)
	Initialized empty Git repository in /big/mybup/
	ok
	# git annex describe mybup "my bup repository at example.com"
	describe mybup ok

Now the remote can be used like any other remote.

	# git annex move my_cool_big_file --to mybup
	move my_cool_big_file (to mybup...)
	Receiving index from server: 1100/1100, done.
	ok

Note that, unlike other remotes, bup does not really support removing
content from its git repositories. This is a feature. :)

	# git annex move my_cool_big_file --from mybup
	move my_cool_big_file (from mybup...)
	  content cannot be removed from bup remote
	failed
	git-annex: 1 failed

See [[special_remotes/bup]] for details.


code:
Another [[special_remote|special_remotes]] that git-annex can use is
a [[special_remotes/ddar]] repository. ddar stores large file contents
in a directory structure of its own, with deduplication. For remote
repositories, ddar requires that ssh is available on the remote, with ddar also
installed remotely. When copying files to the remote, ddar only needs to send
over the network the parts of the files that are not already present remotely.

Unlike bup, ddar uses its own storage format, which allows for both creation
and deletion of de-deduplicated files.

Here's how to create a ddar remote, and describe it.

[[!template id=note text="""
Instead of specifying a remote system, you could choose to make a ddar
remote that is only accessible on the current system, by passing
"ddarrepo=/big/myddar".
"""]]

	# git annex initremote myddar type=ddar encryption=none ddarrepo=example.com:/big/myddar
	initremote ddar (ddar init)
	Initialized empty Git repository in /big/myddar/
	ok
	# git annex describe myddar "my ddar repository at example.com"
	describe myddar ok

Now the remote can be used like any other remote.

	# git annex move my_cool_big_file --to myddar
	move my_cool_big_file (to myddar...)
	ok

See [[special_remotes/ddar]] for details.


code:
So far in this walkthrough, git-annex has been used with a remote
repository on a USB drive. But it can also be used with a git remote
that is truly remote, a host accessed by ssh.

Say you have a desktop on the same network as your laptop and want
to clone the laptop's annex to it:

	# git clone ssh://mylaptop/home/me/annex ~/annex
	# cd ~/annex
	# git annex init "my desktop"

Now you can get files and they will be transferred (using `rsync` via `ssh`):

	# git annex get my_cool_big_file
	get my_cool_big_file (getting UUID for origin...) (from origin...)
	SHA256-s86050597--6ae2688bc533437766a48aa19f2c06be14d1bab9c70b468af445d4f07b65f41e  100% 2159     2.1KB/s   00:00
	ok

When you drop files, git-annex will ssh over to the remote and make
sure the file's content is still there before removing it locally:

	# git annex drop my_cool_big_file
	drop my_cool_big_file (checking origin..) ok

Note that normally git-annex prefers to use non-ssh remotes, like
a USB drive, before ssh remotes. They are assumed to be faster/cheaper to
access, if available. There is a annex-cost setting you can configure in
`.git/config` to adjust which repositories it prefers. See
[[the_man_page|git-annex]] for details.

Also, note that you need full shell access for this to work --
git-annex needs to be able to ssh in and run commands. Or at least,
your shell needs to be able to run the [[git-annex-shell]] command.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawla7u6eLKNYZ09Z7xwBffqLaXquMQC07fU"
 nickname="Matthias"
 subject="Hint for Debian/Ubuntu"
 date="2012-11-07T13:10:15Z"
 content="""
In Debian/Ubuntu the default .bashrc returns immediately if the shell is non-interactive. Make sure to setup the PATH such that it is updated with the location of git-annex-shell before this check! This just cost me an hour of debugging as I didn't notice the return statement early on...
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawniXLhO9mLn-7EDfawdENZ2fQwlGy5w_oc"
 nickname="Michał"
 subject="unrecognized command"
 date="2013-03-12T06:32:56Z"
 content="""
Thanks Matthias, I fought with this as well, this was the tip I needed to move on.  I'm using the Linux standalone, and I had 2 issues getting everything to work without getting git-annex-shell errors.

1. The autoinstalled wrapper could not be found, I had to comment the \"Ubuntu exit\" line and add the $HOME/.ssh to path to get rid of \"command not found\"
2. Had to modify the wrapper by replacing the \"$SSH_ORIGINAL_COMMAND\" by \"$@\" to get rid of \"fatal: unrecognized command ''\"

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 12"
 date="2013-03-12T11:15:11Z"
 content="""
@Michael, the standalone tarball is really meant to run the git-annex assistant. The first time \"git annex webapp\" is run, it will set up the ssh wrapper for you.

I have updated the wrapper to work when ssh is not configured to force a key to run a command.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.81.112"
 subject="comment 2"
 date="2012-05-27T20:53:05Z"
 content="""
When `git annex get` does nothing, it's because it doesn't know a place to get the file from.

This can happen if the `git-annex` branch has not propigated from the place where the file was added.
For example, if on the laptop you had run `git pull ssh master`, that would only pull the master branch, not the git-annex branch.

An easy way to ensure the git-annex branch is kept in sync is to run `git annex sync`
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkaT0B6s9jQuMzQUYRVBgWqtO7BhT_ZSaE"
 nickname="Fernando Seabra"
 subject="cannot get files"
 date="2012-05-27T20:33:09Z"
 content="""
Hi,

I could successfully clone my ssh repo's annex to my laptop, following these instructions.
I'm also able to sync the repositories (laptop and ssh) when I commit new files in the ssh repo.

However, every time I try to get files from the ssh repo (using 'git annex get some_file'), nothing happens.
Do you know what can be happening?

Thanks!
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkaT0B6s9jQuMzQUYRVBgWqtO7BhT_ZSaE"
 nickname="Fernando Seabra"
 subject="comment 3"
 date="2012-05-27T21:13:50Z"
 content="""
Thanks for the quick replay!

I already did 'git annex sync', but it didn't work. The steps were: 'git clone ssh...', then 'cd annex', then 'git annex init \"laptop\"'

After that, I did a 'git annex sync', and tried to get the file, but nothing happens. That's why I found it weird.
Any other thing that might have happened?

Thanks again!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.81.112"
 subject="comment 4"
 date="2012-05-27T21:33:11Z"
 content="""
Try running `git annex whereis` on the file and see where it says it is.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkaT0B6s9jQuMzQUYRVBgWqtO7BhT_ZSaE"
 nickname="Fernando Seabra"
 subject="comment 5"
 date="2012-05-27T21:42:56Z"
 content="""
Hi,

I guess the problem is with git-annex-shell. I tried to do 'git annex get file --from name_ssh_repo', and I got the following:

bash: git-annex-shell: command not found; failed; exit code 127

The same thing happens if I try to do 'git annex whereis'

git-annex-shell is indeed installed. How can I make my shell recognize this command?

Thanks a lot!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.81.112"
 subject="comment 6"
 date="2012-05-27T22:08:50Z"
 content="""
git-annex-shell needs to be installed in the `PATH` on any host that will hold annexed files.

If you installed with cabal, it might be `.cabal/bin/`. Whereever it was installed to is apparently not on the PATH that is set when you ssh into that host.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkaT0B6s9jQuMzQUYRVBgWqtO7BhT_ZSaE"
 nickname="Fernando Seabra"
 subject="comment 7"
 date="2012-05-27T23:35:17Z"
 content="""
Hi,

It was already installed in PATH. In fact, I can call it from the command line, and it is recognized (e.g. calling 'git-annex-shell' gives me 'git-annex-shell: bad parameters'). However, every time I do a 'git annex whereis' or 'git annex get file --from repo', it gives me the following error:

bash: git-annex-shell: command not found
Command ssh [\"-S\",\"/Users/username/annex/.git/annex/ssh/username@example.edu\",\"-o\",\"ControlMaster=auto\",\"-o\",\"ControlPersist=yes\",\"username@example.edu\",\"git-annex-shell 'configlist' '/~/annex'\"] failed; exit code 127

I tried to run this ssh command, but it gives me the same 'command not found' error. It seems that the problem is with the ssh repo?
The ssh repo has a git-annex-shell working and installed in PATH.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmB-gCGEs--zfmvYU-__Hj2FbliUXgxMDs"
 nickname="Jakub"
 subject="Path problems"
 date="2012-07-13T19:15:15Z"
 content="""
Hi,

I have a same 'git-annex-shell command not found' problem as above. I've installed git annex via cabal into my ~/.haskell_bin directory. Then I've added this dir both to ~/.bashrc and ~/.zshrc. I can run git annex or 'git annex-shell' and everything is fine. My guess is that haskell is trying to spawn git-annex-shell with some current $PATH unaware shell like dash maybe?

I've fixed this behavior by using a really ugly hack - I've symlinked ~/.haskell_bin/git-annex-shell to /usr/bin/git-annex-shell on all my machines and the problem is gone. Somehow haskell (or whatever is trying to call git-annex-shell) is unaware of path modifications from .bashrc/.zshrc

Here is the path modification I've used:

export PATH=~/.haskell_bin:$PATH
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmB-gCGEs--zfmvYU-__Hj2FbliUXgxMDs"
 nickname="Jakub"
 subject="Fixed"
 date="2012-07-13T19:27:46Z"
 content="""
Found the problem:

One should never use ~ in such path:

WRONG export PATH=~/somedir:$PATH

Instead one should use $HOME:

GOOD export PATH=$HOME/somedir:$PATH

Can I surpress the message that shell failed with status 255 when a repo is unavailible? I've got two repos pointing to one machine - either via vpn or local lan and I keep getting erros if one is unavailible:

ssh: connect to host 10.9.0.1 port 39882: No route to host
Command ssh [\"-S\",\"/home/pielgrzym/annex/.git/annex/ssh/nas\",\"-o\",\"ControlMaster=auto\",\"-o\",\"ControlPersist=yes\",\"nas\",\"git-annex-shell 'configlist' '/~/annex'\"] failed; exit code 255

"""]]


code:
Like git, git-annex hangs on to every old version of a file (by default),
so you can make tags and branches, and can check them out later to look at
the old files.

	# git tag 1.0
	# rm -f my_cool_big_file
	# git commit -m deleted
	# git checkout 1.0
	# cat my_cool_big_file
	yay! old version still here

Of course, when you `git checkout` an old branch, some old versions of
files may not be locally available, and may be stored in some other
repository. You can use `git annex get` to get them as usual.


code:
#!/bin/sh
# ghci using objects built by cabal
make dist/caballog
$(grep 'ghc --make' dist/caballog | head -n 1 | perl -pe 's/--make/--interactive/; s/.\/[^\.\s]+.hs//; s/-package-id [^\s]+//g; s/-hide-all-packages//; s/-threaded//; s/-O//') $@

path: ghci


code:
Name: git-annex
Version: 5.20140717
Cabal-Version: >= 1.8
License: GPL-3
Maintainer: Joey Hess <joey@kitenet.net>
Author: Joey Hess
Stability: Stable
Copyright: 2010-2014 Joey Hess
License-File: COPYRIGHT
Extra-Source-Files: CHANGELOG
Homepage: http://git-annex.branchable.com/
Build-type: Custom
Category: Utility
Synopsis: manage files with git, without checking their contents into git
Description:
 git-annex allows managing files with git, without checking the file
 contents into git. While that may seem paradoxical, it is useful when
 dealing with files larger than git can currently easily handle, whether due
 to limitations in memory, time, or disk space.
 .
 Even without file content tracking, being able to manage files with git,
 move files around and delete files with versioned directory trees, and use
 branches and distributed clones, are all very handy reasons to use git. And
 annexed files can co-exist in the same git repository with regularly
 versioned files, which is convenient for maintaining documents, Makefiles,
 etc that are associated with annexed files but that benefit from full
 revision control.

Flag S3
  Description: Enable S3 support

Flag WebDAV
  Description: Enable WebDAV support

Flag Inotify
  Description: Enable inotify support

Flag Dbus
  Description: Enable dbus support

Flag Assistant
  Description: Enable git-annex assistant and watch command

Flag Webapp
  Description: Enable git-annex webapp

Flag Webapp-secure
  Description: Secure webapp

Flag Pairing
  Description: Enable pairing

Flag XMPP
  Description: Enable notifications using XMPP

Flag DNS
  Description: Enable the haskell DNS library for DNS lookup

Flag Production
  Description: Enable production build (slower build; faster binary)

Flag Android
  Description: Cross building for Android
  Default: False

Flag AndroidSplice
  Description: Building to get TH splices for Android
  Default: False

Flag TestSuite
  Description: Embed the test suite into git-annex

Flag TDFA
  Description: Use regex-tdfa for wildcards

Flag Feed
  Description: Enable podcast feed support

Flag Quvi
  Description: Enable use of quvi to download videos

Flag Tahoe
  Description: Enable the tahoe special remote

Flag CryptoHash
  Description: Enable use of cryptohash for checksumming

Flag DesktopNotify
  Description: Enable desktop environment notifications

Flag EKG
  Description: Enable use of EKG to monitor git-annex as it runs (at http://localhost:4242/)
  Default: False

Executable git-annex
  Main-Is: git-annex.hs
  Build-Depends: MissingH, hslogger, directory, filepath,
   containers, utf8-string, network (>= 2.0), mtl (>= 2),
   bytestring, old-locale, time, HTTP,
   extensible-exceptions, dataenc, SHA, process, json,
   base (>= 4.5 && < 4.9), monad-control, exceptions (>= 0.5),
   IfElse, text, QuickCheck >= 2.1, bloomfilter, edit-distance, process,
   SafeSemaphore, uuid, random, dlist, unix-compat, async, stm (>= 2.3),
   data-default, case-insensitive
  CC-Options: -Wall
  GHC-Options: -Wall
  Extensions: PackageImports
  -- Some things don't work with the non-threaded RTS.
  GHC-Options: -threaded

  if flag(Production)
    GHC-Options: -O2

  if (os(windows))
    Build-Depends: Win32, Win32-extras
    C-Sources: Utility/winprocess.c
  else
    Build-Depends: unix
    -- Need to list these because they're generated from .hsc files.
    Other-Modules: Utility.Touch Utility.Mounts
    Include-Dirs: Utility
    C-Sources: Utility/libdiskfree.c Utility/libmounts.c
    CPP-Options: -DWITH_CLIBS

  if flag(TestSuite)
    Build-Depends: tasty (>= 0.7), tasty-hunit, tasty-quickcheck, tasty-rerun,
     optparse-applicative, crypto-api
    CPP-Options: -DWITH_TESTSUITE

  if flag(TDFA)
    Build-Depends: regex-tdfa
    CPP-Options: -DWITH_TDFA
  else
    Build-Depends: regex-compat

  if flag(CryptoHash)
    Build-Depends: cryptohash (>= 0.10.0)
    CPP-Options: -DWITH_CRYPTOHASH

  if flag(S3)
    Build-Depends: hS3
    CPP-Options: -DWITH_S3

  if flag(WebDAV)
    Build-Depends: DAV (> 0.6),
     http-client, http-conduit, http-types, lifted-base
    CPP-Options: -DWITH_WEBDAV

  if flag(Assistant) && ! os(solaris)
    CPP-Options: -DWITH_ASSISTANT

  if flag(Assistant)
    if os(linux) && flag(Inotify)
      Build-Depends: hinotify
      CPP-Options: -DWITH_INOTIFY
    else
      if os(darwin)
        Build-Depends: hfsevents
        CPP-Options: -DWITH_FSEVENTS
      else
        if os(windows)
          Build-Depends: Win32-notify
          CPP-Options: -DWITH_WIN32NOTIFY
        else
          if (! os(solaris) && ! os(linux))
            if flag(Android)
              Build-Depends: hinotify
              CPP-Options: -DWITH_INOTIFY
            else
              CPP-Options: -DWITH_KQUEUE
              C-Sources: Utility/libkqueue.c

  if (os(linux))
    if flag(Dbus)
      Build-Depends: dbus (>= 0.10.3)
      CPP-Options: -DWITH_DBUS

    if flag(DesktopNotify)
      if flag(Dbus)
        Build-Depends: dbus (>= 0.10.3), fdo-notify (>= 0.3)
        CPP-Options: -DWITH_DESKTOP_NOTIFY -DWITH_DBUS_NOTIFICATIONS

  if flag(Android)
    Build-Depends: data-endian
    CPP-Options: -D__ANDROID__ -DANDROID_SPLICES -D__NO_TH__
  if flag(AndroidSplice)
    CPP-Options: -DANDROID_SPLICES

  if flag(Webapp)
    Build-Depends:
     yesod, yesod-default, yesod-static, yesod-form, yesod-core,
     http-types, transformers, wai, wai-extra, warp, warp-tls,
     blaze-builder, crypto-api, hamlet, clientsession,
     template-haskell, data-default, aeson, path-pieces,
     shakespeare
    CPP-Options: -DWITH_WEBAPP
  if flag(Webapp) && flag (Webapp-secure)
    Build-Depends: warp-tls (>= 1.4), securemem, byteable
    CPP-Options: -DWITH_WEBAPP_SECURE

  if flag(Pairing)
    Build-Depends: network-multicast, network-info
    CPP-Options: -DWITH_PAIRING

  if flag(XMPP) && (! os(windows))
    Build-Depends: network-protocol-xmpp, gnutls (>= 0.1.4), xml-types
    CPP-Options: -DWITH_XMPP

  if flag(DNS)
    Build-Depends: dns
    CPP-Options: -DWITH_DNS

  if flag(Feed)
    Build-Depends: feed
    CPP-Options: -DWITH_FEED

  if flag(Quvi)
    Build-Depends: aeson
    CPP-Options: -DWITH_QUVI

  if flag(Tahoe)
    Build-Depends: aeson
    CPP-Options: -DWITH_TAHOE

  if flag(EKG)
    Build-Depends: ekg
    GHC-Options: -with-rtsopts=-T
    CPP-Options: -DWITH_EKG

source-repository head
  type: git
  location: git://git-annex.branchable.com/


code:
{- git-annex main program dispatch
 -
 - Copyright 2010-2014 Joey Hess <joey@kitenet.net>
 -
 - Licensed under the GNU GPL version 3 or higher.
 -}

{-# LANGUAGE CPP #-}

import System.Environment (getArgs, getProgName)
import System.FilePath

import qualified CmdLine.GitAnnex
import qualified CmdLine.GitAnnexShell
#ifdef WITH_TESTSUITE
import qualified Test
#endif

#ifdef mingw32_HOST_OS
import Utility.UserInfo
import Utility.Env
import Config.Files
import System.Process
import System.Exit
#endif

main :: IO ()
main = do
	ps <- getArgs
	run ps =<< getProgName
  where
	run ps n
		| isshell n = CmdLine.GitAnnexShell.run ps
		| otherwise =
#ifdef mingw32_HOST_OS
			winEnv gitannex ps
#else
			gitannex ps
#endif
	gitannex ps =
#ifdef WITH_TESTSUITE
		case ps of
			("test":ps') -> Test.main ps'
			_ -> CmdLine.GitAnnex.run ps
#else
		CmdLine.GitAnnex.run ps
#endif
	isshell n = takeFileName n == "git-annex-shell"

#ifdef mingw32_HOST_OS
{- On Windows, if HOME is not set, probe it and set it.
 - This is a workaround for some Cygwin commands needing HOME to be set,
 - and for there being no known way to set environment variables on
 - Windows, except by passing an environment in each call to a program.
 - While ugly, this workaround is easier than trying to ensure HOME is set
 - in all calls to the affected programs.
 -
 - If TZ is set, unset it.
 - TZ being set can interfere with workarounds for Windows timezone
 - horribleness, and prevents getCurrentTimeZone from seeing the system
 - time zone.
 -
 - Due to Windows limitations, have to re-exec git-annex with the new
 - environment.
 -}
winEnv :: ([String] -> IO ()) -> [String] -> IO ()
winEnv a ps = do
	e <- getEnvironment
	home <- myHomeDir
	let e' = wantedenv e home
	if (e' /= e)
		then do
			cmd <- readProgramFile
			(_, _, _, pid) <- createProcess (proc cmd ps)
				{ env = Just e' }
			exitWith =<< waitForProcess pid
		else a ps
  where
	wantedenv e home = delEntry "TZ" $ case lookup "HOME" e of
		Nothing -> e
		Just _ -> addEntries
			[ ("HOME", home)
			, ("CYGWIN", "nodosfilewarning")
			] e
#endif


code:
{- git-union-merge program
 -
 - Copyright 2011 Joey Hess <joey@kitenet.net>
 -
 - Licensed under the GNU GPL version 3 or higher.
 -}

import System.Environment

import Common
import qualified Git.UnionMerge
import qualified Git.Config
import qualified Git.CurrentRepo
import qualified Git.Branch
import qualified Git.Index
import qualified Git

header :: String
header = "Usage: git-union-merge ref ref newref"

usage :: IO a
usage = error $ "bad parameters\n\n" ++ header

tmpIndex :: Git.Repo -> FilePath
tmpIndex g = Git.localGitDir g </> "index.git-union-merge"

setup :: Git.Repo -> IO ()
setup = cleanup -- idempotency

cleanup :: Git.Repo -> IO ()
cleanup g = nukeFile $ tmpIndex g

parseArgs :: IO [String]
parseArgs = do
	args <- getArgs
	if length args /= 3
		then usage
		else return args

main :: IO ()
main = do
	[aref, bref, newref] <- map Git.Ref <$> parseArgs
	g <- Git.Config.read =<< Git.CurrentRepo.get
	_ <- Git.Index.override $ tmpIndex g
	setup g
	Git.UnionMerge.merge aref bref g
	_ <- Git.Branch.commit "union merge" newref [aref, bref] g
	cleanup g