code:
Let's say you want to distribute some big files to the whole world.
You can of course, just drop them onto a website. But perhaps you'd like to
use git-annex to manage those files. And as an added bonus, why not let
anyone in the world clone your site and use `git-annex get`!

My site like this is [downloads.kitenet.net](https://downloads.kitenet.net).
Here's how I set it up. --[[Joey]]

1. Set up a web site. I used Apache, and configured it to follow symlinks.
   `Options FollowSymLinks`
2. Put some files on the website. Make sure it works.
3. `git init; git annex init`
4. `git config core.sharedrepository world` (Makes sure files
   are always added with permissions that allow everyone to read them.)
5. We want users to be able to clone the git repository over http, because
   git-annex can download files from it over http as well. For this to
   work, `git update-server-info` needs to get run after commits. The
   git `post-update` hook will take care of this, you just need to enable
   the hook. `chmod +x .git/hooks/post-update`
6. `git annex add; git commit -m added`
7. Make sure users can still download files from the site directly.
8. Instruct advanced users to clone a http url that ends with the "/.git/"
   directory. For example, for downloads.kitenet.net, the clone url
   is `https://downloads.kitenet.net/.git/`
9. Set up a git `post-receive` hook to update the repository's working tree
   when changes are pushed to it. See below for details.

When users clone over http, and run git-annex, it will
automatically learn all about your repository and be able to download files
right out of it, also using http.

## post-receive hook

If you have git-annex 4.20130703, the post-receive hook mentioned above
in step 9 just needs to run `git annex merge`.

With older versions of git-annex, you can instead use `git annex sync`.

There are two gotchas with some versions of git to be aware of when writing
this post-receive hook.

1. The hook may be run with the current directory set to the `.git`
   directory, and not the top of your work tree. So you need to `cd ..` or
   similar in the hook.
2. `GIT_DIR` may be set to `.`, which will not be right after changing
   directory. So you will probably want to unset it.

Here's a post-receive hook that takes these problems into account:

<pre>
#!/bin/sh
unset GIT_DIR
cd ..
git annex merge
</pre>


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnYLUTs4jFpBPwOlIIJ6qD8xdqZPboJafM"
 nickname="Oluf"
 subject="Combine this with a 'public' repository-group"
 date="2013-07-16T09:44:26Z"
 content="""
Hi,

would it be possible to do this whith the contents of a public repository-group (a non-bare public repository)?
"""]]

repo_name: kakafarm/git

license: unknown


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.90"
 subject="comment 2"
 date="2013-07-16T17:54:28Z"
 content="""
You can choose which files get stored in the public repository, and are thus accessible to the public.
However, note that since the git repository is published, anyone could clone it and see all the names and hashes of your files, even if you've not pushed the file contents to the public repository.

Currently the way the \"public\" [[repository group|preferred_content]] works only makes it be usable with special remotes. This is because it uses a `preferreddir` setting in the special remote configuration.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnRRUUZo3W7pAoRoST8P_l0PtUBhvYuzDg"
 nickname="Lyle"
 subject="web storage"
 date="2014-01-17T22:59:27Z"
 content="""
Is there a low cost web hosting solution that would support a public git-annex repo relatively simply with simple access to download the public files.

I figure I could set up an Amazon EC2 micro instance and mount an s3 share, hosting the git-annex remote, but this is a lot of overhead for something that dropbox does with 1 click \"share dropbox link\"?

Any suggestions would be great!
"""]]


code:
Scenario
========

You have a server where you want to welcome other people to push files, say for a family photo album. People have their own user account, so by default they will not be able to read/write from each other's repositories, due to git-annex strict restrictions.

Solution
========

Setup a shared git repository:

    git init shared ; cd shared # you can also do this on an existing git annex repo
    git config core.sharedrepository group
    chmod g+rwX -R .
    chown -R :media .

The idea here is to use the new (since [[news/version 4.20130909]]) support for git's `sharedRepository` configuration and restrict access to a specific group (instead of the default, a single user). You can also this to make the files accessible to all users on the system:

    git config core.sharedrepository world
    chmod a+rwX -R .

This will make sure that you anyone can operate that git annex repository remotely.

Third party applications
------------------------

Now if another application that is not aware of git's `sharedRepository` configuration (say a [[bittorrent]] daemon) writes files there, you may want to make sure that the files created are also writable by everyone. This is more tricky, but one way of doing this is with the [[!wikipedia setgid]] bit:

    find -type d -exec chmod g+s {} \;

You will also need to start the process with a proper [[!wikipedia umask]] (`002` instead of `022`).

(!) I haven't actually tested this part. --[[anarcat]]

See also
========

 * [[tips/setup a public repository on a web site]]
 * [[news/version 4.20130909]]
 * [[bugs/acl not honoured in rsync remote]]: why this does not work on encrypted remotes


code:
[[!comment format=mdwn
 username="madduck"
 ip="2001:a60:f0fb:0:224:d7ff:fe04:c82c"
 subject="Does not work"
 date="2014-04-06T10:48:06Z"
 content="""
This does not seem to work, even with latest git-annex. I think it's because git-annex [[!debbug desc=\"creates files without honouring +s on the parent directory\" 729757]]. I just found out it also doesn't honour default ACLs. I hope this can be fixed soon.
"""]]


code:
skydriveannex 0.2.1
=========

Hook program for gitannex to use [skydrive](http://en.wikipedia.org/wiki/SkyDrive) (previously *Windows Live SkyDrive* and *Windows Live Folders*) as backend

# Requirements:

    python2
    python-yaml

Credit for the Skydrive api interface goes to https://github.com/mk-fg/python-skydrive

# Install
Clone the git repository in your home folder.

    git clone git://github.com/TobiasTheViking/skydriveannex.git

This should make a ~/skydriveannex folder

# Setup
Make the file executable, and link it into PATH

    cd ~/skydriveannex; chmod +x git-annex-remote-skydrive; sudo ln -sf `pwd`/git-annex-remote-skydrive /usr/local/bin/git-annex-remote-skydrive

# Commands for gitannex:

    git annex initremote skydrive type=external externaltype=skydrive encryption=shared folder=gitannex

An oauth authentication link should now be launched in the default browser. Authenticate, and use the last url as OAUTH key.

    OAUTH='URL after last redirect' git annex initremote skydrive type=external externaltype=skydrive encryption=shared folder=gitannex
    git annex describe skydrive "the skydrive library"


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkAUMhKOSkh9JaBA6xst3XxQIIsDEq5Zd4"
 nickname="Ovidiu"
 subject="Syncing not working :-/"
 date="2014-05-17T19:52:33Z"
 content="""
Error log says:

    git-annex-remote-skydrive is not installed in PATH (/Applications/git-annex.app/Contents/MacOS/bundle:/Applications/git-annex.app/Contents/MacOS/bundle:/Applications/git-annex.app/Contents/MacOS/bundle:/Applications/git-annex.app/Contents/MacOS/bundle:/Applications/git-annex.app/Contents/MacOS/bundle:/Applications/git-annex.app/Contents/MacOS/bundle:/Applications/git-annex.app/Contents/MacOS/bundle:/Applications/git-annex.app/Contents/MacOS/bundle:/usr/bin:/bin:/usr/sbin:/sbin)

BUT I did a symlink, I even did 2 to 2 different locations that are in my path:

    cd /Users/ovi/skydriveannex/
    ln -sf `pwd`/git-annex-remote-skydrive /usr/bin/git-annex-remote-skydrive
    ln -sf `pwd`/git-annex-remote-skydrive /opt/local/bin/git-annex-remote-skydrive
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkAUMhKOSkh9JaBA6xst3XxQIIsDEq5Zd4"
 nickname="Ovidiu"
 subject="working for git-annex assistant?"
 date="2014-05-13T18:40:31Z"
 content="""
Will following these steps lead to the Skydrive repo being visible and recognized by the assistant too?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 2"
 date="2014-05-16T16:26:50Z"
 content="""
The assistant will use any repositories you set up, no matter how you set them up.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkAUMhKOSkh9JaBA6xst3XxQIIsDEq5Zd4"
 nickname="Ovidiu"
 subject="Hitting trouble"
 date="2014-05-16T17:49:59Z"
 content="""
    shiny-2:skydriveannex ovi$ git annex initremote skydrive type=external externaltype=skydrive encryption=shared folder=git-annex
    git-annex: First run: git-annex init
    shiny-2:skydriveannex ovi$

Above wasn't mentioned in the tutorial so I tried it:

    shiny-2:skydriveannex ovi$ git-annex init
    init  ok
    (Recording state in git...)
    shiny-2:skydriveannex ovi$

What now? Never got asked about any OAUTH...
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkAUMhKOSkh9JaBA6xst3XxQIIsDEq5Zd4"
 nickname="Ovidiu"
 subject="comment 4"
 date="2014-05-16T17:51:29Z"
 content="""
And no link was opened automatically
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 5"
 date="2014-05-16T19:03:34Z"
 content="""
You have to run git-annex init before you can use other git-annex commands. I think if you run git annex initremote now, it will work.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkAUMhKOSkh9JaBA6xst3XxQIIsDEq5Zd4"
 nickname="Ovidiu"
 subject="comment 6"
 date="2014-05-16T19:25:03Z"
 content="""
tried it but no luck:

    git annex initremote skydrive type=external externaltype=skydrive encryption=shared folder=git-annex
    initremote skydrive (encryption setup) (shared cipher) env: python2: No such file or directory
    git-annex: external special remote protocol error, unexpectedly received \"\" (unable to parse command)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkAUMhKOSkh9JaBA6xst3XxQIIsDEq5Zd4"
 nickname="Ovidiu"
 subject="making some progress"
 date="2014-05-17T19:08:16Z"
 content="""
Figured out how to solve that:
    which python
    /opt/local/bin/python
    shiny-2:git-annex ovi$ sudo ln -s `which python` /usr/bin/python2

The URL opened in my browser, I allowed access, waited for the redirect, copied the URL, hit the next problem:

    OAUTH='https://login.live.com/oauth20_desktop.srf?code=hidden_on_purpose' git annex initremote skydrive type=external externaltype=skydrive encryption=shared folder=git-annex
initremote skydrive (encryption setup) (shared cipher)  21:06:27 [skydriveannex-0.2.1] <module> : 'Starting'
     21:06:27 [skydriveannex-0.2.1] login : 'EXCEPTION api.auth_get_token:ImportError('No module named requests',)'
    git-annex: You need to set OAUTH environment variables and folder and encryption parameters when running initremote.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkAUMhKOSkh9JaBA6xst3XxQIIsDEq5Zd4"
 nickname="Ovidiu"
 subject="Done! Success!"
 date="2014-05-17T19:23:06Z"
 content="""
realized I need requests. to install I need pip.

    sudo port install py27-pip

    pip install requests

    login : 'EXCEPTION api.auth_get_token:ImportError('No module named six',)'

    pip install six

Finished! Seems to be working and I can see this repository in my assistant!
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkAUMhKOSkh9JaBA6xst3XxQIIsDEq5Zd4"
 nickname="Ovidiu"
 subject="QUESTION"
 date="2014-05-17T19:29:37Z"
 content="""
Wondering about this:

    encryption=shared

I read what it means: http://git-annex.branchable.com/encryption/

But now I'm wondering how the repos which I set up via the assistant have been setup. when setting them up I only had the choice between \"encrypt all files\" and \"disable encryption\" so how can I check what settings my other repos are using?
"""]]


code:
Suppose you have a USB thumb drive and are using it as a git annex
repository. You don't trust the drive, because you could lose it, or
accidentally run it through the laundry. Or, maybe you have a drive that
you know is dying, and you'd like to be warned if there are any files
on it not backed up somewhere else. Maybe the drive has already died
or been lost.

You can let git-annex know that you don't trust a repository, and it will
adjust its behavior to avoid relying on that repositories's continued
availability.

	# git annex untrust usbdrive
	untrust usbdrive ok

Now when you do a fsck, you'll be warned appropriately:

	# git annex fsck .
	fsck my_big_file
	  Only these untrusted locations may have copies of this file!
	  	05e296c4-2989-11e0-bf40-bad1535567fe  -- portable USB drive
	  Back it up to trusted locations with git-annex copy.
	failed

Also, git-annex will refuse to drop a file from elsewhere just because
it can see a copy on the untrusted repository.

It's also possible to tell git-annex that you have an unusually high
level of trust for a repository. See [[trust]] for details.


code:
Amazon Glacier provides low-cost storage, well suited for archiving and
backup. But it takes around 4 hours to get content out of Glacier.

Recent versions of git-annex support Glacier. To use it, you need to have
[glacier-cli](http://github.com/basak/glacier-cli) installed.

First, export your Amazon AWS credentials:

        # export AWS_ACCESS_KEY_ID="08TJMT99S3511WOZEP91"
        # export AWS_SECRET_ACCESS_KEY="s3kr1t"

Now, create a gpg key, if you don't already have one. This will be used
to encrypt everything stored in Glacier, for your privacy. Once you have
a gpg key, run `gpg --list-secret-keys` to look up its key id, something
like "2512E3C7"

Next, create the Glacier remote.

	# git annex initremote glacier type=glacier keyid=2512E3C7
	initremote glacier (encryption setup with gpg key C910D9222512E3C7) (gpg) ok

The configuration for the Glacier remote is stored in git. So to make another
repository use the same Glacier remote is easy:

        # cd /media/usb/annex
        # git pull laptop
        # git annex enableremote glacier
        initremote glacier (gpg) ok

Now the remote can be used like any other remote.

        # git annex move my_cool_big_file --to glacier
        copy my_cool_big_file (gpg) (checking glacier...) (to glacier...) ok

But, when you try to get a file out of Glacier, it'll queue a retrieval
job:

	# git annex get my_cool_big_file
	get my_cool_big_file (from glacier...) (gpg)
	glacier: queued retrieval job for archive 'GPGHMACSHA1--862afd4e67e3946587a9ef7fa5beb4e8f1aeb6b8'
	  Recommend you wait up to 4 hours, and then run this command again.
	failed

Like it says, you'll need to run the command again later. Let's remember to
do that:

	# at now + 4 hours
	at> git annex get my_cool_big_file

Another oddity of Glacier is that git-annex is never entirely sure
if a file is still in Glacier. Glacier inventories take hours to retrieve,
and even when retrieved do not necessarily represent the current state.

So, git-annex plays it safe, and avoids trusting the inventory:

	# git annex copy important_file --to glacier
	copy important_file (gpg) (checking glacier...) (to glacier...) ok
	# git annex drop important_file
	drop important_file (gpg) (checking glacier...)
	  Glacier's inventory says it has a copy.
	  However, the inventory could be out of date, if it was recently removed.
	  (Use --trust-glacier if you're sure it's still in Glacier.)

	(unsafe)
	  Could only verify the existence of 0 out of 1 necessary copies

Like it says, you can use `--trust-glacier` if you're sure
Glacier's inventory is correct and up-to-date.

A final potential gotcha with Glacier is that glacier-cli keeps a local
mapping of file names to Glacier archives. If this cache is lost, or
you want to retrieve files on a different box than the one that put them in
glacier, you'll need to use `glacier vault sync` to rebuild this cache.

See [[special_remotes/Glacier]] for details.


code:
[[!comment format=mdwn
 username="http://grossmeier.net/"
 nickname="greg"
 subject="missing steps?"
 date="2014-02-18T23:51:00Z"
 content="""
I setup a glacier remote on one machine and it successfully created the vault and is syncing files to it.

One another machine, after git-annex sync'ing, I did:

[[!format sh \"\"\"
greg@x200s:~/Photos$ git-annex enableremote glacier
enableremote glacier
  Set both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to use glacier
git-annex: Failed creating glacier vault.
\"\"\"]]

So then I try:
[[!format sh \"\"\"
greg@x200s:~/Photos$ AWS_ACCESS_KEY_ID=HAHA AWS_SECRET_ACCESS_KEY=NOPE git-annex --debug enableremote glacier
[2014-02-18 15:43:56 PST] read: git [\"--git-dir=/home/greg/Photos/.git\",\"--work-tree=/home/greg/Photos\",\"show-ref\",\"git-annex\"]
[2014-02-18 15:43:56 PST] read: git [\"--git-dir=/home/greg/Photos/.git\",\"--work-tree=/home/greg/Photos\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
[2014-02-18 15:43:56 PST] read: git [\"--git-dir=/home/greg/Photos/.git\",\"--work-tree=/home/greg/Photos\",\"log\",\"refs/heads/git-annex..8108714116d08f93aa427b9ddced48cd5f2b4b72\",\"--oneline\",\"-n1\"]
[2014-02-18 15:43:56 PST] read: git [\"--git-dir=/home/greg/Photos/.git\",\"--work-tree=/home/greg/Photos\",\"log\",\"refs/heads/git-annex..742ba908f791e440a6cc85073ef505a96dd66aa4\",\"--oneline\",\"-n1\"]
[2014-02-18 15:43:56 PST] read: git [\"--git-dir=/home/greg/Photos/.git\",\"--work-tree=/home/greg/Photos\",\"log\",\"refs/heads/git-annex..071487394544a20253a70ada4ea71fcc28f9fc13\",\"--oneline\",\"-n1\"]
[2014-02-18 15:43:56 PST] read: git [\"--git-dir=/home/greg/Photos/.git\",\"--work-tree=/home/greg/Photos\",\"log\",\"refs/heads/git-annex..c8aecc22da7b84bbb82f083ce783cc699cef1c67\",\"--oneline\",\"-n1\"]
[2014-02-18 15:43:56 PST] chat: git [\"--git-dir=/home/greg/Photos/.git\",\"--work-tree=/home/greg/Photos\",\"cat-file\",\"--batch\"]
enableremote glacier [2014-02-18 15:43:56 PST] call: glacier [\"--region=us-west-2\",\"vault\",\"create\",\"glacier-7e5c0010-2634-4a5e-bc7b-6fea84b8b947\"]
git-annex: Failed creating glacier vault.
\"\"\"]]

What am I missing?

Also, why is it trying to *create* the valut? It's already there with content in it!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.172"
 subject="comment 2"
 date="2014-02-20T19:24:09Z"
 content="""
@greg, the only thing you might have missed is the need to use `glacier vault sync` to build a cache if enabling the glacier remote in another place. And that whole issue with it needing a local cache may mean few people are using glacier with more than one repository accessing the remote.

However, this sounds like a bug. There is a comment in the source code that \"glacier vault create will succeed even if the vault already exists.\" .. perhaps it has changed since that was written. Or perhaps the command failed for some other reason, I don't know.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.172"
 subject="comment 3"
 date="2014-02-20T19:34:48Z"
 content="""
I've changed it to avoid running glacier value create when enabling an existing glacier remote. Hopefully that fixes it.
"""]]


code:
[[!comment format=mdwn
 username="http://grossmeier.net/"
 nickname="greg"
 subject="comment 4"
 date="2014-02-20T22:04:58Z"
 content="""
Along with stupid python problems which are now fixed (all my fault, and hopefully didn't cause more noise here than needed), the only thing that didn't go as stated was:

[[!format sh \"\"\"
greg@x200s:~/Photos$ git-annex enableremote glacier
enableremote glacier
  Set both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to use glacier
git-annex: Failed creating glacier vault.
greg@x200s:~/Photos$ AWS_ACCESS_KEY_ID=lolno AWS_SECRET_ACCESS_KEY=lolno git-annex enableremote glacier
enableremote glacier ok
(Recording state in git...)
greg@x200s:~/Photos$
\"\"\"]]

The guide says that info is sync'd.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.172"
 subject="good"
 date="2014-02-21T15:20:39Z"
 content="""
If you mean the creds are not remembered, that's controlled by the embedcreds= option to initremote, and it only defaults to embedding them for gacier when using strong encryption (not encryption=shared).
"""]]


code:
[[!comment format=mdwn
 username="http://grossmeier.net/"
 nickname="greg"
 subject="confirmed"
 date="2014-02-21T17:53:02Z"
 content="""
Yeah, I choose no encryption for this one for worst case scenario reasons (I still want photos of my kid even if I loss my gpg key and my house burns down). Now about setting up http://git.kitenet.net/?p=gpg.git;a=blob;f=README.sss;hb=HEAD ......
"""]]


code:
git-annex extends git's usual remotes with some [[special_remotes]], that
are not git repositories. This way you can set up a remote using say,
Amazon S3, and use git-annex to transfer files into the cloud.

First, export your Amazon AWS credentials:

	# export AWS_ACCESS_KEY_ID="08TJMT99S3511WOZEP91"
	# export AWS_SECRET_ACCESS_KEY="s3kr1t"

Now, create a gpg key, if you don't already have one. This will be used
to encrypt everything stored in S3, for your privacy. Once you have
a gpg key, run `gpg --list-secret-keys` to look up its key id, something
like "2512E3C7"

Next, create the S3 remote, and describe it.

	# git annex initremote cloud type=S3 keyid=2512E3C7
	initremote cloud (encryption setup with gpg key C910D9222512E3C7) (checking bucket) (creating bucket in US) (gpg) ok
	# git annex describe cloud "at Amazon's US datacenter"
	describe cloud ok

The configuration for the S3 remote is stored in git. So to make another
repository use the same S3 remote is easy:

	# cd /media/usb/annex
	# git pull laptop
	# git annex initremote cloud
	initremote cloud (gpg) (checking bucket) ok

Now the remote can be used like any other remote.

	# git annex copy my_cool_big_file --to cloud
	copy my_cool_big_file (gpg) (checking cloud...) (to cloud...) ok
	# git annex move video/hackity_hack_and_kaxxt.mov --to cloud
	move video/hackity_hack_and_kaxxt.mov (checking cloud...) (to cloud...) ok

See [[special_remotes/S3]] for details.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnoUOqs_lbuWyZBqyU6unHgUduJwDDgiKY"
 nickname="Matt"
 subject="ANNEX_S3 vs AWS for keys"
 date="2012-05-29T12:24:25Z"
 content="""
The instructions state ANNEX_S3_ACCESS_KEY_ID and ANNEX_SECRET_ACCESS_KEY but git-annex cannot connect with those constants. git-annex tells me to set both \"AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY\" instead, which works. This is with Xubuntu 12.04.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.81.112"
 subject="comment 2"
 date="2012-05-29T19:10:42Z"
 content="""
Thanks, I've fixed that. (You could have too.. this is a wiki ;)
"""]]


code:
[[!comment format=mdwn
 username="annexuser"
 ip="64.71.7.82"
 subject="Altering AWS credentials"
 date="2014-04-15T21:59:43Z"
 content="""
If I revoke old AWS credentials and create new ones, how would I inform git-annex of the change to `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.191"
 subject="comment 4"
 date="2014-04-17T19:44:55Z"
 content="""
You can use `git annex enableremote` to change an existing remote's configuration. So this should work:

    # export AWS_ACCESS_KEY_ID=\"newRANDOMGOBBLDEYGOOK\"
    # export AWS_SECRET_ACCESS_KEY=\"news3kr1t\"
    # git annex enableremote cloud

"""]]


code:
[Google Cloud Storage](https://cloud.google.com/products/cloud-storage)
supports the same API as Amazon S3, so the
[[S3 special remote|special_remotes/S3]] can be used with it.
Here is a configuration example:

	git annex initremote cloud type=S3 encryption=none host=storage.googleapis.com port=80

Thanks to jterrance for the [original tip](https://gist.github.com/4576324).
--[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnaH44G3QbxBAYyDwy0PbvL0ls60XoaR3Y"
 nickname="Nigel"
 subject="AWS credentials"
 date="2013-05-31T10:23:23Z"
 content="""
This looks very valuable - Google are offering a free 5GB up until 2013 June 30th

Sign in here[1] get your credentials here[2] under the section “Interoperable Access” (Source[3])

    # Set up AWS credentials
    $ export AWS_ACCESS_KEY_ID=\"YOUR-KEY\"
    $ export AWS_SECRET_ACCESS_KEY=\"YOUR-SECRET\"

1. http://gs-signup-redirect.appspot.com/ -or- https://developers.google.com/storage/docs/signup
2. https://storage.cloud.google.com/m
3. http://fog.io/storage/
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlPfB0_GcGJ827kZqRE1PF4D23rxpfOeg4"
 nickname="Björn"
 subject="Can't find developer API key and secret"
 date="2014-06-01T21:25:54Z"
 content="""
According to the last comment, and this page:
* https://developers.google.com/storage/docs/migrating#keys

there's supposed to be a \"Interoperable Access\" menu option, something I can't find. Note that both the previous comment and this page refers to Googles old Developer Console.

I can find a few things under \"APIs and Auth\" => \"Credentials\" under the heading \"Public API access\", but no matter what kind of key I try to create I can't get anything other than an API key. No secrets here, and no \"Show\"-button for newly created keys.

Another thing worth mentioning is that these API keys I'm seeing do not like described in that first link in this comment. It's much longer, contains both upper- and lowercase and also contains dashes.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlPfB0_GcGJ827kZqRE1PF4D23rxpfOeg4"
 nickname="Björn"
 subject="comment 3"
 date="2014-06-17T19:18:09Z"
 content="""
I just found the \"Interoperable access\" thing. The instructions from Google (linked in previous comments) are not perfect.

After step 4 click \"Google Cloud Storage\", then at the bottom you have a heading called \"Interoperable Access\" where you have to activate it for this project.

Then it appears at the bottom of the left hand menu.
"""]]


code:
[Box.com](http://box.com/) is a file storage service, currently notable
for providing 50 gb of free storage if you sign up with its Android client.
(Or a few gb free otherwise.)

git-annex can use Box as a [[special remote|special_remotes]].
Recent versions of git-annex make this very easy to set up:

	WEBDAV_USERNAME=you@example.com WEBDAV_PASSWORD=xxxxxxx git annex initremote box.com type=webdav url=https://dav.box.com/dav/git-annex chunk=50mb encryption=shared

Note the use of [[chunking]]; Box has a 100 mb maximum file size, and this
breaks up large files into chunks before that limit is reached.

# old davfs2 method

This method is deprecated, but still documented here just in case.
Note that the files stored using this method cannot reliably be retreived
using the webdav special remote.

## davfs2 setup

* First, install
  the [davfs2](http://savannah.nongnu.org/projects/davfs2) program,
  which can mount Box using WebDAV. On Debian, just `sudo apt-get install davfs2`
* Allow users to mount davfs filesystems, by ensuring that
  `/sbin/mount.davfs` is setuid root. On Debian, just `sudo dpkg-reconfigure davfs2`
* Add yourself to the davfs2 group.

        sudo adduser $(whoami) davfs2

* Edit `/etc/fstab`, and add a line to mount Box using davfs.

        sudo mkdir -p /media/box.com
        echo "https://dav.box.com/dav/	/media/box.com	davfs	noauto,user	0 0" | sudo tee -a /etc/fstab

* Create `~/.davfs2/davfs2.conf` with some important settings:

        mkdir ~/.davfs2/
        echo use_locks 0 > ~/.davfs2/davfs2.conf
        echo cache_size 1 >> ~/.davfs2/davfs2.conf
        echo delay_upload 0 >> ~/.davfs2/davfs2.conf

* Create `~/.davfs2/secrets`. This file contains your Box.com login and password.
  Your login is probably the email address you signed up with.

        echo "/media/box.com joey@kitenet.net mypassword" > ~/.davfs2/secrets
        chmod 600 ~/.davfs2/secrets

* Now you should be able to mount Box, as a non-root user:

        mount /media/box.com

## git-annex setup

You need git-annex version 3.20120303 or newer, which adds support for chunking
files larger than Box's 100 mb limit.

Create the special remote, in your git-annex repository.
** This example is non-encrypted; fill in your gpg key ID for a securely
encrypted special remote! **

	git annex initremote box.com type=directory directory=/media/box.com chunk=2mb encryption=none

Now git-annex can copy files to box.com, get files from it, etc, just like
with any other special remote.

	% git annex copy bigfile --to box.com
	bigfile (to box.com...) ok
	% git annex drop bigfile
	bigfile (checking box.com...) ok
	% git annex get bigfile
	bigfile (from box.com...) ok


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlmvi8Z-rUen8xIety5uiJitI5rnMrz8Jk"
 nickname="Timothy"
 subject="updated url for drive mount"
 date="2013-12-18T20:20:12Z"
 content="""
I had to change the url in fstab to https://**dav**.box.com/dav
"""]]


code:
## Intro

This tip is based on my (Matt Ford) experience of using `git annex` with my out-and-about netbook which hits many different wifi networks and has no fixed home or address.

I'm not using a bare repository that allows pushing (an alternative solution) nor do I fancy allowing `git push` to run against my desktop checked out repository (perhaps I worry over nothing?)

None of this is really `git annex` specific but I think it is useful to know...

## Dealing with no fixed hostname

Essentially set up two repos as per the [[walkthrough]].

Desktop as follows:

    cd ~/annex
    git init
    git annex init "desktop"

And the laptop like this

    git clone ssh://desktop/annex
    git init
    git annex init "laptop"

Now we want to add the the repos as remotes of each other.

For the laptop it is easy:

    git remote add desktop ssh://desktop/~/annex

However for the desktop to add an ever changing laptops hostname it's a little tricky.  We make use of remote SSH tunnels to do this.  Essentially we have the laptop (which always knows it's own name and address and knows the address of the desktop) create a tunnel starting on an arbitrary port at the desktop and heads back to the laptop on it's own SSH server port (22).

To do this make part of your laptop's SSH config look like this:

    Host desktop
    User matt
    HostName desktop.example.org
    RemoteForward 2222 localhost:22

Now on the desktop to connect over the tunnel to the laptop's SSH port you need this:

    Host laptop
    User matt
    HostName localhost
    port 2222

So to add the desktop's remote:

a) From the laptop ensure the tunnel is up

    ssh desktop

b) From the desktop add the remote

    git remote add laptop ssh://laptop/~/annex

So now you can work on the train, pop on the wifi at work upon arrival, and sync up with a `git pull && git annex get`.

An alternative solution may be to use direct tunnels over Openvpn.


code:
[[!comment format=mdwn
 username="http://adamspiers.myopenid.com/"
 nickname="Adam"
 subject="comment 1"
 date="2011-12-23T13:31:33Z"
 content="""
ControlPersist is awesome - thanks!

Here's [an alternative, git-specific approach](http://thread.gmane.org/gmane.comp.version-control.home-dir/502).
"""]]


code:
[Gitolite](https://github.com/sitaramc/gitolite) is a git repository
manager. Here's how to add git-annex support to gitolite, so you can
`git annex copy` files to a gitolite repository, and `git annex get`
files from it.

A nice feature of using gitolite with git-annex is that users can be given
read-only access to a repository, and this allows them to `git annex get`
file contents, but not change anything.

First, you need new enough versions:

* the current `master` branch of gitolite works with git-annex (tested 2014-04-19),
  but v3.5.3 and earlier v3.x require use of the `git-annex` branch.
* gitolite 2.2 also works -- this version contains a git-annex-shell ADC
  and supports "ua" ADCs.
* git-annex 3.20111016 or newer needs to be installed on the gitolite
  server. Don't install an older version, it wouldn't be secure!

### Instructions for gitolite `master` branch

To setup gitolite to work with git-annex, you can follow the instructions on the gitolite website,
and just add `'git-annex-shell ua',` to the ENABLE list in `~/.gitolite.rc`.

Here are more detailed instructions:

1: Create a `git` user

sudo adduser \
   --system \
   --shell /bin/bash \
   --gecos 'git version control' \
   --group \
   --disabled-password \
   --home /home/git git

2: Copy a public SSH key for the user you want to be the gitolite administrator.
In the instructions below, I placed the key in a file named `/home/git/me.pub`.

3: Clone and install gitolite

First switch to the `git` user (e.g. `sudo su - git`) and then run:

cd
git clone https://github.com/sitaramc/gitolite.git
mkdir -p bin
./gitolite/install -ln

4: Add `~/bin` to `PATH`

Make sure that `~/bin` is in the `PATH`, since that's where gitolite installed its binary.  Do something like this:

echo 'export PATH=/home/git/bin:$PATH' >> .profile
export PATH=/home/git/bin:$PATH

5: Configure gitolite

Edit `~/.gitolite.rc` to enable the git-annex-shell command.
Find the `ENABLE` list and add this line in there somewhere:

'git-annex-shell ua',

Now run gitolite's setup:

gitolite setup -pk me.pub
rm me.pub

### Instructions for gitolite 2.2

And here's how to set it up. The examples are for gitolite as installed
on Debian with apt-get, but the changes described can be made to any
gitolite installation, just with different paths.

Set `$GL_ADC_PATH` in `.gitolite.rc`, if you have not already done so.

echo '$GL_ADC_PATH = "/usr/local/lib/gitolite/adc/";' >>~gitolite/.gitolite.rc

Make the ADC directory, and a "ua" subdirectory.

mkdir -p /usr/local/lib/gitolite/adc/ua

Install the git-annex-shell ADC into the "ua" subdirectory from the gitolie repository.

cd /usr/local/lib/gitolite/adc/ua/
cp gitolite/contrib/adc/git-annex-shell .

Now all gitolite repositories can be used with git-annex just as any
ssh remote normally would be used. For example:

# git clone gitolite@localhost:testing
Cloning into testing...
Receiving objects: 100% (18/18), done.
# cd testing
# git annex init
init  ok
# cp /etc/passwd my-cool-big-file
# git annex add my-cool-big-file
add my-cool-big-file ok
(Recording state in git...)
# git commit -m added
[master d36c8b4] added
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 120000 my-cool-big-file
# git push --all
Counting objects: 17, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (12/12), done.
Writing objects: 100% (14/14), 1.39 KiB, done.
Total 14 (delta 0), reused 1 (delta 0)
To gitolite@localhost:testing
   c552a38..db4653e  git-annex -> git-annex
   29cd204..d36c8b4  master -> master
# git annex copy --to origin
copy my-cool-big-file (checking origin...) (to origin...)
WORM-s2502-m1318875140--my-cool-big-file
        2502 100%    0.00kB/s    0:00:00 (xfer#1, to-check=0/1)

sent 2606 bytes  received 31 bytes  1758.00 bytes/sec
total size is 2502  speedup is 0.95
ok

### Troubleshooting

I got an error like this when setting up gitolite *after* setting up a local git repo and git annex:

git-annex-shell: First run: git-annex init
Command ssh ["git@git.example.com","git-annex-shell 'configlist' '/~/myrepo.git'"] failed; exit code 1

because I forgot to "git push --all" after adding the new gitolite remote.


code:
[[!comment format=mdwn
 username="http://ertai.myopenid.com/"
 nickname="npouillard"
 subject="git-annex no longer supported by gitolite g3"
 date="2013-03-25T12:47:21Z"
 content="""
See http://gitolite.com/gitolite/dev-status.html for some details.
"""]]


code:
[[!comment format=mdwn
 username="http://mildred.fr/"
 nickname="mildred"
 subject="Problems with URL ending with ".git""
 date="2013-05-24T12:15:16Z"
 content="""
Hi,

I noticed using the git-annex branch of gitolite v3 that the same URL with \".git\" at the end would not work in git-annex. For example my test repository was `git@git2.mildred.fr:u/mildred/Annex.git` but it didn't work until I converted it to `git@git2.mildred.fr:u/mildred/Annex`

On the server, the repository is in `repositories/u/mildred/Annex.git`

If I try a copy with git-annex for example, I would get:

    $ git annex copy titi --to test
    copy titi (checking test...) FATAL: u/mildred/Annex.git mildred DENIED

    (unable to check test) failed
    git-annex: copy: 1 failed

(test is the name of my remote and titi is my file)

Note, in my gitolite conf, I have:

    repo u/CREATOR/[a-zA-Z0-9].*
      C    = @all
      RW+D = CREATOR
      RW   = WRITERS
      R    = READERS

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn5RcmefXjrl1vmbHIiOWQyXGXVKxlm3rg"
 nickname="Kavin"
 subject="comment 12"
 date="2013-07-25T03:20:15Z"
 content="""
latest code of gitolite does not support git-annex ? I could not find a way to make it work ?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm9tgeFE5v-arAYYftSv3yUTI5Q4qB2C9M"
 nickname="Khaije"
 subject="git-annex with gitolite FTW"
 date="2013-08-13T15:13:07Z"
 content="""
The steps to activate git-annex integration have changed/simplified for v3.

1) during install, be sure to use the 'git-annex' branch, rather than master[fn:1].

2) to enable git-annex-shell, open ~/.gitolite.rc and insert 'git-annex-shell' => 'ua' into the hash list in the COMMANDS array.[fn:2]
#'git-annex-shell'   =>  'ua',

[fn:1] We'd like to have this feature-branch merged to master, so please send Sitaram feedback, positive and negative, based on your experiences.
[fn:2] There is no GL_ADC_PATH and no \"ua\" subdirectory here, and nothing to \"install\"; the command now comes with gitolite.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSbvo_NbY-ev1VKtzwo7nEqUmvRO6rXGA"
 nickname="François"
 subject="comment 14"
 date="2013-09-22T18:30:45Z"
 content="""
@khaije

Could you paste your config file? Here is mine: http://paste.debian.net/44856/
I don't have any COMMANDS array. Could you elaborate your modifications please?

Thanks.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnRaueN1AcM8pIMofH5-wQ1Kr4T0GBe8sA"
 nickname="wayne"
 subject="git-annex with gitolite-rc"
 date="2013-10-19T17:03:32Z"
 content="""
@François

The proper array in .gitolite.rc seems to be the \"ENABLE\" array, which it appears is parsed into the COMMANDS array in src/lib/Gitolite/Rc.pm

@Khaije

Using 'git-annex-shell' => 'ua' doesn't seem to work for me. The program still fails in src/gitolite-shell around line 163 (gitolite repo version b1d3c05):

    _die \"suspicious characters loitering about '$soc'\"
      if $rc{COMMANDS}{ words[0] } ne 'ua' and $soc !~ $REMOTE_COMMAND_PATT;

When I insert $rc{COMMANDS}{ words[0] } into the _die message, it shows up as \"1\" instead of \"ua\" as I was expecting.

When I manually set $rc{COMMANDS}{ words[0] } to 'ua' slightly earlier in the script, the git-annex-shell command gets run but it seems to fail to parse the result of the configlist command properly because then I get

    Failed to get annex.uuid configuration of repository origin

    Instead, got: \"annex.uuid=\ncore.gcrypt=\n\"

I am actually about to give up on the notion of using git-annex and gitolite together. Maybe. I am interested to know if anyone else is having similar problems.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmbo-yMzW_lkBrS4ICNn5XMr8saYtI1_WY"
 nickname="Douglas"
 subject="Any further info regarding gitolite support?"
 date="2013-11-14T03:20:31Z"
 content="""
We have reached the same point as the previous poster from 25 days ago.
$ git annex copy --to origin
FATAL: suspicious characters loitering about 'git-annex-shell 'configlist' '/~/testing''

  Remote origin does not have git-annex installed; setting remote.origin.annex-ignore
git-annex: cannot determine uuid for origin

Anyone actually have this working?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm9tgeFE5v-arAYYftSv3yUTI5Q4qB2C9M"
 nickname="Khaije"
 subject="comment 17"
 date="2013-11-23T02:14:12Z"
 content="""
my gitolite.rc is available at https://gist.github.com/khaije1/7609848

For whatever reason I've found this to be very simple to get working so I'd guess there's a missing ingredient somewhere. The combination of gitolite and git-annex is valuable to me so I'll add documents to the url above in hopes it will assist some people with getting the same value.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawll4Kgp7nMuOKdB0FfbcYZ3KRq7HCS0Slc"
 nickname="Laura"
 subject="Using Gitolite 3.2"
 date="2014-01-17T20:14:16Z"
 content="""
Adding:

 'git-annex-shell' =>1,

To the .gitolite.rc file resulted in the \"FATAL: suspicous characters loitering about 'git-annex-shell 'configlist' '/~/testing''...

Gitolite source code (https://github.com/sitaramc/gitolite/commit/b1d3c0571409b7c6279fc6a77253c3bc262ab425#diff-79a3701e9e2cee0ea1316451c21a3fec) requires this entry:

'git-annex-shell ua'

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnftLWVOF0DUdXr2HSW4IKzuqtW8V7X3YU"
 nickname="Adrian"
 subject="How can the git annex init command be called on the server?"
 date="2014-03-25T06:58:56Z"
 content="""
The latest commit in the gitolite repository \"git-annex support, finally in master!\" looks really promissing. I'm currently using ubuntu trusty with updated gitolite3 package and the configuration provided by Khaije. One line needs to be changed: 'git-annex-shell' => 'ua', instead of 'git-annex-shell' =>1,

However, one little detail is still open for me. I need to call the server side \"git annex init\" on the server (sudo su gitolite3; cd ~/repositories/testing.git && git annex init). I cannot find a way to initialize the server from client side. E.g. git annex init && git push --all is not enough. The man page describes git annex initremote for other server types but not for gitolite remotes. Wouldn't we need something similar for gitolite as well? Or is there a better solution which I do not recognize?

"""]]


code:
[[!comment format=mdwn
 username="http://www.openid.albertlash.com/openid/"
 ip="71.178.29.218"
 subject="comment 1"
 date="2011-12-24T06:08:45Z"
 content="""
Looks like you are missing a closing double quote on the line:

echo '$GL_ADC_PATH = \"/usr/local/lib/gitolite/adc/;' >>~gitolite/.gitolite.rc

right after /;

I got this working by the way - great stuff.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.41"
 subject="comment 20"
 date="2014-03-26T18:24:30Z"
 content="""
@Adrian, that's good news. It would be helpful if someone could update the top of this page to document how to use git-annex with the new version of gitolite.
This is a wiki..

You're not supposed to need to use initremote when dealing with normal git remotes. It seems that something got lost that automatically initialize the remote repository in this situation. I've fixed it so that it will be set up when `git-annex-shell configlist` is run.

(@wayne, this fixes the problem you reported too..)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn26WQjIP5fnMgQF_L_k3Q3UrR5v8mjRTY"
 nickname="Ellis"
 subject="Worked for me, updated wiki"
 date="2014-04-19T09:41:11Z"
 content="""
I tried the `master` branch of gitolite today on my server, and it works fine!  Updated the wiki with the commands I used to setup gitolite.
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 2"
 date="2011-12-24T16:54:31Z"
 content="""
I've fixed the typo (anyone can edit pages in this wiki FWIW.)
"""]]


code:
[[!comment format=mdwn
 username="bremner"
 ip="156.34.79.193"
 subject="repo name conventions?"
 date="2011-12-30T21:41:13Z"
 content="""
I'm confused by the fact that the git-annex-shell adc rejects any repo names that don't start with /~/ since none of my repos start that way. It seems work ok if I just delete /\~ from the front of the regex, but I feel like I must be missing something.
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 4"
 date="2011-12-31T00:29:45Z"
 content="""
Well a repo url like `gitolite@localhost:testing` puts it in the gitolite user's /~/testing

This worked when I added the gitolite stuff, anyway.. Let's see if it still does:

<pre>
joey@gnu:~/tmp>mkdir g
joey@gnu:~/tmp>cd g
joey@gnu:~/tmp/g>git init
Initialized empty Git repository in /home/joey/tmp/g/.git/
joey@gnu:~/tmp/g>git annex init
init  ok
joey@gnu:~/tmp/g>git remote add test 'gitolite@localhost:testing'
joey@gnu:~/tmp/g>touch foo
joey@gnu:~/tmp/g>git annex add foo
add foo (checksum...) ok
(Recording state in git...)
joey@gnu:~/tmp/g>git annex copy foo --to test --debug
git [\"--git-dir=/home/joey/tmp/g/.git\",\"--work-tree=/home/joey/tmp/g\",\"ls-files\",\"--cached\",\"-z\",\"--\",\"foo\"]
git [\"--git-dir=/home/joey/tmp/g/.git\",\"--work-tree=/home/joey/tmp/g\",\"check-attr\",\"annex.numcopies\",\"-z\",\"--stdin\"]
git [\"--git-dir=/home/joey/tmp/g/.git\",\"--work-tree=/home/joey/tmp/g\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
git [\"--git-dir=/home/joey/tmp/g/.git\",\"--work-tree=/home/joey/tmp/g\",\"show-ref\",\"git-annex\"]
git [\"--git-dir=/home/joey/tmp/g/.git\",\"--work-tree=/home/joey/tmp/g\",\"cat-file\",\"--batch\"]
Running: ssh [\"-4\",\"gitolite@localhost\",\"git-annex-shell 'configlist' '/~/testing'\"]
</pre>

Still seems right, the ADC's regexp will match this the git-annex shell command.
"""]]


code:
[[!comment format=mdwn
 username="bremner"
 ip="156.34.79.193"
 subject="gitolite gets different paths for different urls"
 date="2011-12-31T01:50:49Z"
 content="""
I guess there is some path rewriting going in in gitolite proper because if try a url of the form
ssh://git@localhost/testing, then it still works with gitolite, but fails with the ADC because
the repo is passed as /testing:

Running: ssh [\"git@host\",\"git-annex-shell 'configlist' '/recommend'\"]
Running: ssh [\"git@host\",\"git-annex-shell 'configlist' '/recommend'\"]

What I have to ask Sitaram and or find in the docs is if this is a bug or a feature in gitolite.  I can see how the leading slash would get swallowed up by this line

$repo = \"'$REPO_BASE/$repo.git'\"

in gl-auth-command, but I guess that isn't the whole story.
"""]]


code:
[[!comment format=mdwn
 username="bremner"
 ip="156.34.79.193"
 subject="ssh://gitolite-host/repo-name is supposed to work"
 date="2011-12-31T03:34:17Z"
 content="""
I confirmed with Sitaram that this is intentional, if probably under-documented.
Since the ADC strips the leading /~/ in assigning $start anyway, I guess something like the following will  work
<pre>

diff --git a/contrib/adc/git-annex-shell b/contrib/adc/git-annex-shell
index 7f9f5b8..523dfed 100755
--- a/contrib/adc/git-annex-shell
+++ b/contrib/adc/git-annex-shell
@@ -28,7 +28,7 @@ my $cmd=$ENV{SSH_ORIGINAL_COMMAND};
 # the second parameter.
 # Further parameters are not validated here (see below).
 die \"bad git-annex-shell command: $cmd\"
-    unless $cmd =~ m#^(git-annex-shell '\w+' ')/\~/([0-9a-zA-Z][0-9a-zA-Z._\@/+-
+    unless $cmd =~ m#^(git-annex-shell '\w+' ')/(?:\~\/)?([0-9a-zA-Z][0-9a-zA-Z.
 my $start = $1;
 my $repo = $2;
 my $end = $3;
</pre>
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 7"
 date="2011-12-31T18:32:28Z"
 content="""
That patch seems ok, it doesn't seem to allow through any repo locations that were blocked before.

So, it has my blessing.. but the ADC is in gitolite and will need to be patched there.
"""]]


code:
[[!comment format=mdwn
 username="bremner"
 ip="156.34.79.193"
 subject="afaict git annex normalizes urls on the client side."
 date="2011-12-31T22:29:38Z"
 content="""
After some debugging printing, here is my current understanding.

- urls of the form git@host:~repo or ssh://git@host

  - git sends commands like  \"git-receive-pack '~/repo'
  - gitolite converts these to $REPO_BASE/~/repo which fails.  ~/repo would also fail fwiw.
  - git-annex sends seems /~/repo, which works

- urls of the form git@host:/repo or ssh://git@host/repo

  - git sends \"git-receive-pack '/db/cs3383'\"
  - gitolite converts this to $REPO_BASE/repo which works
  - git annex sends \"git-annex-shell 'inannex' '/repo' ...\" which works, but only with the patch above.

- urls of the form git@host:repo

  - git sends \"git-receive-pack 'repo'
  - gitolite converts this to $REPO_BASE/repo, which works
  - git-annex sends \"git-annex-shell 'inannex' '/~/db/cs3383'...\", which also works for git-annex-shell.

So the weird case is the last one where git and git-annex are sending different things over the wire.
I don't know if you have other motivations for doing the url normalization on the client side, but it isn't needed for gitolite, and in some sense complicates things a little.  On the other hand, now that I see what is going on, it isn't a big deal to just strip the leading /~ off in the adc. It does lead to the odd situation of some URLs working for git-annex but not git.
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 9"
 date="2012-01-02T16:27:55Z"
 content="""
Ah right. git-annex normalizes all git ssh style user@host:dir to valid uris, which is where the `/~/` comes from. I don't anticipate this changing on the git-annex side.
"""]]


code:
A handy alternative to the default [[backend|backends]] is the
SHA1 backend. This backend provides more git-style assurance that your data
has not been damaged. And the checksum means that when you add the same
content to the annex twice, only one copy need be stored in the backend.

The only reason it's not the default is that it needs to checksum
files when they're added to the annex, and this can slow things down
significantly for really big files. To make SHA1 the default, just
add something like this to `.gitattributes`:

	* annex.backend=SHA1


code:
The web can be used as a [[special_remote|special_remotes]] too.

	# git annex addurl http://example.com/video.mpeg
	addurl example.com_video.mpeg (downloading http://example.com/video.mpeg)
	########################################################## 100.0%
	ok

Now the file is downloaded, and has been added to the annex like any other
file. So it can be renamed, copied to other repositories, and so on.

To add a lot of urls at once, just list them all as parameters to
`git annex addurl`.

## trust issues

Note that git-annex assumes that, if the web site does not 404, and has the
right file size, the file is still present on the web, and this counts as
one [[copy|copies]] of the file. If the file still seems to be present
on the web, it will let you remove your last copy, trusting it can be
downloaded again:

	# git annex drop example.com_video.mpeg
	drop example.com_video.mpeg (checking http://example.com/video.mpeg) ok

If you don't [[trust]] the web to this degree, just let git-annex know:

	# git annex untrust web
	untrust web ok

With the result that it will hang onto files:

	# git annex drop example.com_video.mpeg
	drop example.com_video.mpeg (unsafe)
	  Could only verify the existence of 0 out of 1 necessary copies
	  Also these untrusted repositories may contain the file:
	  	00000000-0000-0000-0000-000000000001  -- web
	  (Use --force to override this check, or adjust numcopies.)
	failed

## attaching urls to existing files

You can also attach urls to any file already in the annex:

	# git annex addurl --file my_cool_big_file http://example.com/cool_big_file
	addurl my_cool_big_file ok
	# git annex whereis my_cool_big_file
	whereis my_cool_big_file (2 copies)
  	00000000-0000-0000-0000-000000000001 -- web
   	27a9510c-760a-11e1-b9a0-c731d2b77df9 -- here

## configuring filenames

By default, `addurl` will generate a filename for you. You can use
`--file=` to specify the filename to use.

If you're adding a bunch of related files to a directory, or just don't
like the default filenames generated by `addurl`, you can use `--pathdepth`
to specify how many parts of the url are put in the filename.
A positive number drops that many paths from the beginning, while a negative
number takes that many paths from the end.

	# git annex addurl http://example.com/videos/2012/01/video.mpeg
	addurl example.com_videos_2012_01_video.mpeg (downloading http://example.com/videos/2012/01/video.mpeg)
	# git annex addurl http://example.com/videos/2012/01/video.mpeg --pathdepth=2
	addurl 2012_01_video.mpeg (downloading http://example.com/videos/2012/01/video.mpeg)
	# git annex addurl http://example.com/videos/2012/01/video.mpeg --pathdepth=-2
	addurl 01_video.mpeg (downloading http://example.com/videos/2012/01/video.mpeg)

## videos

There's support for downloading videos from sites like YouTube, Vimeo,
and many more. This relies on [quvi](http://quvi.sourceforge.net/) to find
urls to the actual videos files.

When you have quvi installed, you can just
`git annex addurl http://youtube.com/foo` and it will detect that
it is a video and download the video content for offline viewing.

Later, in another clone of the repository, you can run `git annex get` on
the file and it will also be downloaded with the help of quvi. This works
even if the video host has transcoded or otherwise changed the video
in the meantime; the assumption is that these video files are equivilant.

There is an `annex.quvi-options` configuration setting that can be used
to pass parameters to quvi. For example, you could set `git config
annex.quvi-options "--format low"` to configure it to download low
quality videos from YouTube.

Note that for performance reasons, the url is not checked for redirects,
so some shortened urls will not be detected. You can
either load the short url in a browser to get the full url, or you
can force use of quvi with redirect detection, by prepending "quvi:" to the
url.

To download a youtube playlist, you need to find the feed associated with that
playlist, and pass it to `git annex importfeed`. There does not seem to be
an easy link anywhere to get the feed, but you can construct its url
manually. For a playlist like
"https://www.youtube.com/playlist?list=PL4F80C7D2DC8D9B6C", the
feed is "http://gdata.youtube.com/feeds/api/playlists/PL4F80C7D2DC8D9B6C"

More details about youtube feeds at <http://googlesystem.blogspot.com/2008/01/youtube-feeds.html>
-- `git-annex importfeed` should handle all of them.

## podcasts

This is done using `git annex importfeed`. See [[downloading podcasts]].


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlc1og3PqIGudOMkFNrCCNg66vB7s-jLpc"
 nickname="Paul"
 subject="can addurl use hashing once the file is downloaded?"
 date="2012-09-20T21:01:30Z"
 content="""
There are resources that I want to add to my annex that are currently available
via a URL, but it seems like if I add these using `git-annex addurl`, they get
symlinked to file in the annex/objects directory that starts with `URL-...`,
instead of the more typical `SHA256-...`, and this does not change even after
the files are downloaded.

My concern is that I really want to ensure that these files don't change, which
is the appeal of content-addressable symlinking of normal files (as opposed to
URL addressable ones).

Would there be a way to automate the injection of hash-based symlinking for
files that are added via addurl? Sometimes I add a bunch of files via ``addurl
--fast``, and after I've download them via ``get``, it would be nice to have
those files have the same level of data integrity as when I download them using
something outside of git-annex, add them to the annex, and do an ``addurl
--file`` afterward.

Thanks for all of your hard work!

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 subject="comment 2"
 date="2012-09-20T21:55:57Z"
 content="""
`addurl` only uses the URL- keys if you run it with --fast. Otherwise it downloads the content and hashes it the same as `add` does.

If you use `--fast`, you can go back and `git annex migrate` the file once it's been downloaded, to convert
it to the SHA backend.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnx8kHW66N3BqmkVpgtXDlYMvr8TJ5VvfY"
 nickname="Yaroslav"
 subject="how to drop one of the urls?"
 date="2013-04-12T14:53:29Z"
 content="""
is there a way to remove one of the urls? e.g. if I have

    $> git annex whereis fail2ban_logo.png
    whereis fail2ban_logo.png (1 copy)
      	00000000-0000-0000-0000-000000000001 -- web

      web: http://www.fail2ban.org/fail2ban_logo.png
      web: http://www.onerussian.com/tmp/statsmodes.png
    ok

and would like to remove the fail2ban.org one... ?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 4"
 date="2013-04-22T21:28:03Z"
 content="""
You can use `git annex rmurl $file $url`, which I just added to git-annex.

(Also, `git annex drop $file --from web` will remove all the urls..)
"""]]


code:
[[!comment format=mdwn
 username="Xyem"
 ip="87.194.19.134"
 subject="comment 5"
 date="2014-04-04T15:25:39Z"
 content="""
Adding videos from youtube ends up with it using the URL backend, even without fast.

    $ git init quvitest
    $ cd quvitest/
    $ git annex init
    $ git annex addurl https://www.youtube.com/watch?v=mghhLqu31cQ
    (... file is downloaded ...)
    $ find .git/annex/objects/ -type f
    .git/annex/objects/1J/Wp/URL--quvi&chttps&c%%www.youtube.com%watch,63v,61mghhLqu31cQ/URL--quvi&chttps&c%%www.youtube.com%watch,63v,61mghhLqu31cQ

Is migrating manually required or should I log a bug?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.244"
 subject="comment 6"
 date="2014-04-07T20:07:45Z"
 content="""
Using the URL backend for youtube is intentional. Youtube may serve up different encodings for the same video over time, and this way git-annex treats them all as equvilant. If you want to \"freeze\" the repository to the current one, use `git annex migrate`, and be prepared for `git annex get --from web` to not work long term.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnkXKIAfgPsuV-EJpXm3jCc3q9IZwzvCcw"
 nickname="Garrett"
 subject="Default pathdepth"
 date="2014-05-08T21:36:00Z"
 content="""
Is there away to change the default pathdepth so I do not need to add --pathdepth=-1 everything I run addurl?
"""]]


code:
[Gource](http://code.google.com/p/gource/) is an amazing animated
visualisation of a git repository.

Normally, gource shows files being added, removed, and changed in
the repository, and the user(s) making the changes. Of course it can be
used in this way in a repository using git-annex too; just run `gource`.

The other way to use gource with git-annex is to visualise the movement of
annexed file contents between repositories. In this view, the "users" are
repositories, and they move around the file contents that are being added
or removed from them with git-annex.

[[!img screenshot.jpg]]

To use gource this way, first go into the directory you want to visualize,
and use `git annex log` to make an input file for `gource`:

	git annex log --gource | tee gource.log
	sort gource.log | gource --log-format custom -

The `git annex log` can take a while, to speed it up you can use something
like `--after "4 months ago"` to limit how far back it goes.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm9IQStaE1el95_9s77CgmJhxZwCwUeN9A"
 nickname="Sam"
 subject="You can also use it for LIVE display now"
 date="2013-12-19T14:29:18Z"
 content="""
hi, check out http://puffingdev.com/live-gource-visually-display-your-svngit-activity-live/

You can use it to display LIVE updates of a git repo, nice to have on a screen in your company's common area for example.
"""]]


code:
A git-annex repository on a removable USB drive is great, until the cable
falls out at the wrong time and git's repository gets trashed. The way
git checksums everything and the poor quality of USB media makes this
perhaps more likely than you would expect. If this happens to you,
here's a way to recover that makes the most of whatever data is left
on the drive.

* First, run `git fsck`. If it does not report any problems, your data
  is fine, and you don't need to proceed further.
* So `git fsck` says the git repository is corrupted. But probably the data
  git-annex stored is fine. Your first step is to clone another copy
  of the git repository from somewhere else. Let's call this clone
  "$good", and the corrupted repository "$bad".
* Preserve your git configuration changes, and the `annex.uuid` setting:
  `mv $bad/.git/config $good/.git/config`
* Move annexed data into the new repository: `mkdir $good/.git/annex; mv
  $bad/.git/annex/objects $good/.git/annex/objects`
* Reinitalize git-annex: `cd $good; git annex init`
* Check for any problems with the annexed data: `cd $good; git annex fsck`
* Now you can remove the corrupted repository, the new one is ready to use.

Alternatively, recent versions of git-annex have a `git annex repair`
command that uses <http://git-repair.branchable.com/> to repair a
repository in-place. The git-annex assistant will detect most corruptions
and offer to run the repair for you automatically.

--[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnq-RfkVpFN15SWvQ2lpSGAi0XpNQuLxKM"
 nickname="Yuval"
 subject="Automation"
 date="2014-04-27T11:34:44Z"
 content="""
Could this step be automated?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmZilYULa6CDEGfuagoDlesyakBgnf-dF8"
 nickname="Maarten"
 subject="comment 2"
 date="2014-05-23T23:50:58Z"
 content="""
What if the drive is destroyed?  How would I re-initialize a new drive to act as a substitute for the remote that was just lost?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.36"
 subject="comment 3"
 date="2014-05-24T18:29:03Z"
 content="""
@Maarten git clone from one of the other clones of the repository to the new drive. Also, recent versions of git-annex have a `git annex reinit` command that can be used to easily initialize a repository with the same settings as a repository that was lost.
"""]]


code:
So you lost a thumb drive containing a git-annex repository. Or a hard
drive died or some other misfortune has befallen your data.

Unless you configured backups, git-annex can't get your data back. But it
can help you deal with the loss.

Go somewhere that knows about the lost repository, and mark it as
dead:

	git annex dead usbdrive

This retains the [[location_tracking]] information for the repository,
but avoids trying to access it, or list it as a location where files
are present.

If you later found the drive, you could let git-annex know it's found
like so:

	git annex semitrust usbdrive


code:
[[!comment format=mdwn
 username="http://dlaxalde.myopenid.com/"
 nickname="dl"
 subject="comment 1"
 date="2012-05-31T14:36:33Z"
 content="""
Is there a way to have git-annex completely ignore a repository? I see that
the `dead` command adds the uuid of the repository to `trust.log` but does
not change `uuid.log`. Is it enough to remove the corresponding line in
`uuid.log` and `trust.log`?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.8.243"
 subject="comment 3"
 date="2012-05-31T17:01:37Z"
 content="""
`dead` is the best we can do. The automatic merging used on the git-annex branch tends to re-add lines that are deleted in one repo when merging with another that still has them.
"""]]


code:
[[!comment format=mdwn
 username="https://openid.stackexchange.com/user/e65e6d0e-58ba-41de-84cc-1f2ba54cf574"
 nickname="Mica Semrick"
 subject="Update?"
 date="2013-12-16T00:31:27Z"
 content="""
Is it reasonable to update this page to include `git-annex forget`?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="comment 4"
 date="2013-12-16T17:04:44Z"
 content="""
I wouldn't recommend running `git annex forget --drop-dead` right after losing a drive. You might find the drive again, or get the dead drive to work again. Even if you know you'll never find it, you may want to query git-annex to find files that were only located on that drive, and deal with them, etc.

But if you're worried about that information in the git-annex branch cluttering up the git repository, you keep in mind that `git annex forget` will let you remove it at some point in the future.
"""]]


code:
[[!comment format=mdwn
 username="https://openid.stackexchange.com/user/e65e6d0e-58ba-41de-84cc-1f2ba54cf574"
 nickname="Mica Semrick"
 subject="forget command"
 date="2013-12-17T05:52:52Z"
 content="""
Unfortunately, I plugged in the drive and it was alive just long enough to clone the repo and pull down about half of the data. Then it just started clicking. I did my best to wipe the drive before I returned it, but it didn't even make it that far.

Now I've got a replacement drive. Since my system auto-mounts based on the partition label; I had been using the label to identify external drives. So I want to describe the drive the same way to my system & git-annex.

In this case, I think I'd like to completely get rid of any notion the repos have of the old drive.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="comment 6"
 date="2013-12-17T16:11:52Z"
 content="""
You could use `git annex forget --drop-dead` in that situation, but what I would do is first `git annex describe lostremote \"my old drive that I lost\"` and then set up the replacement drive with whatever description you had before.
"""]]


code:
[[!comment format=mdwn
 username="http://alan.petitepomme.net/"
 nickname="Alan Schmitt"
 subject="still need to remove the remote after a "dead""
 date="2014-03-07T18:03:47Z"
 content="""
Hello,

I decommissioned a machine and installed git annex on another machine, with the same IP as the initial one. To avoid errors, I declared the first repository as dead and created a new one (with a new remote). However, I also had to remove the git remote to the dead machine as \"git annex sync\" kept telling me synchronizing to this dead machine failed (the ssh succeeded, using the name automatically generated by git annex, but the repository was not at the same path on the new machine, so it could not find it).

Should a dead repository not tried to be synchronized at all during git annex sync?

Thanks,

Alan
"""]]


code:
Here's the annex-du script that I use:

#!/bin/sh
git annex find "$@" --include '*' --format='${bytesize}\n' |awk '{ sum += $1; nfiles++; } END { printf "%d files, %.3f MB\n", nfiles, sum/1000000 } '

This one can be slow on a large number of files, but it has an advantage of being able to use all of the filtering available in git annex find.
For example, to figure out how much is stored in remote X, do

annex-du --in=X


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="comment 1"
 date="2013-07-12T19:36:28Z"
 content="""
Ah, I just found that git annex info can do the same :)
Disregard this.
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/2grhJvAC049fJnvALDXek.6MRZMTlg--#eec89"
 nickname="John"
 subject="comment 2"
 date="2013-08-30T06:09:29Z"
 content="""
You may want to try my `sizes` tool on Hackage.  Just pass `-A` and it will be aware of the annex and report sizes as if no files were annexed.  The only downside is that it reports file usage for replicated content multiple times, as if you'd copied the data out of the annex rather than hardlinked all duplicate copies (although, this may be exactly the behavior some people want).
"""]]


code:
This is git-annex's todo list. Link items to [[todo/done]] when done. A more complete [[design/roadmap/]] is also available.

[[!inline pages="./todo/* and !./todo/done and !link(done)
and !*/Discussion" actions=yes postform=yes show=0 archive=yes]]


code: One problem I keep having when using a direct repo is that in order to get to the previous versions of a file you have to convert that repo to indirect and then checkout previous commits this becomes problematic when the repo in question is large conversion takes a long time and applications gets confused if there are open files from the repo as they go from actual files to symlinks. Is it possible to have a separate annex command that will checkout a previous version of a file to a different directory so we can replace/inspect it.


code:
[[!comment format=mdwn
 username="zardoz"
 ip="92.227.51.179"
 subject="comment 1"
 date="2014-05-13T20:25:22Z"
 content="""
I’ve added a bug on basically the same issue.
http://git-annex.branchable.com/bugs/Revert_to_old_file_version_in_direct_mode___40__VFAT__41__/

One problem is that if your direct-mode repo is on VFAT, you can’t even switch to indirect mode temporarily.
"""]]


code:
[[!comment format=mdwn
 username="zardoz"
 ip="92.227.51.179"
 subject="comment 2"
 date="2014-05-17T14:18:31Z"
 content="""
It occured to me that one could do something like that by «git show
earlier-commit:file > filename». A problem with this is that a
subsequent «get» will get the old content, but won’t put it in place
until invoking «fsck».

"""]]


code:
[[!comment format=mdwn
 username="zardoz"
 ip="92.227.51.179"
 subject="comment 3"
 date="2014-05-17T14:24:20Z"
 content="""
Plus it seems that the file can’t be dropped normally afterwards.
"""]]


code:
I like the way btsync search for the peer. So if I need to sync my laptop and other family laptop both with a total different and changing network setup the two device found each other do NAT traversal if needed use relays but the end the two folders are synced. But its closed-source :( I like git and git-annex looks really great. I'm learning it now.

First I thought with xmpp I can sync files without ssh/rsync or other remote access to my devices just with two jabber account. Now I know I can't. :(

It would be just great to have some means to sync files without cloud just the two device. Without the ssh / rsync jut share some secret and the devices do the rest. :-o

Anyway thanks for hearing. I'm looking forward to know more about git-annex. Thank you for that sw. =-<>-=

> [[design/assistant/telehash]] --[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkEHjHAWnJ0BJzdv_hePwU1my8X4wCseh8"
 nickname="Sz"
 subject="comment 1"
 date="2013-07-23T11:00:21Z"
 content="""
Why not use xmpp for file transfer too not only for git sync?
"""]]


code:
[[!comment format=mdwn
 username="http://johan.kiviniemi.name/"
 ip="83.145.237.224"
 subject="comment 2"
 date="2013-07-23T11:37:53Z"
 content="""
Transferring the data over XMPP would almost certainly take too much XMPP server bandwidth, but using something like [libjingle](https://developers.google.com/talk/libjingle/) to set up P2P connections should work nicely. That would require libjingle (or equivalent) bindings for Haskell, though. Libjingle negotiates over XMPP to set up the P2P connection and provides a TCP-like layer for reliable, ordered communication. One could use that for both git metadata and the file transfers.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkEHjHAWnJ0BJzdv_hePwU1my8X4wCseh8"
 nickname="Sz"
 subject="comment 3"
 date="2013-07-24T09:08:45Z"
 content="""
I'm for it! +1
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 4"
 date="2013-08-03T07:12:40Z"
 content="""
NAT traversal requires central infrastructure and is unreliable at best. Obviously, a for-profit entity like bittorrent can shoulder that.

For LAN situations, [zeroconf](http://www.zeroconf.org/) along with passphrase-based pairing may be the long-term answer. Arguably, that would enhance vanilla Git almost as much as it would enhance git-annex, but that does not seem likely to happen.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm5iosFbL2By7UFeViqkc6v-hoAtqILeDA"
 nickname="Laszlo"
 subject="comment 5"
 date="2013-08-25T07:48:18Z"
 content="""
What is the problem with bittorrent protocol in general?
It is some technicality or purely philosophical?

Best,
 Laszlo

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8"
 nickname="develop"
 subject="comment 6"
 date="2013-08-25T08:39:15Z"
 content="""
I just did a cursory search on haskell torrent support. And the required pieces do seem to be be there.
https://github.com/jlouis/combinatorrent or https://github.com/astro/haskell-torrent for downloading. i'm not sure if either supports DHT, but that exists here https://github.com/aninhumer/haskell-dht

That said, i think implementing this would require some quite major overhauls in the system. It probably won't be trivial to implement.

Note: This is for straight \"bittorrent\", not for \"bittorrent sync\". Bittorrent sync is closed source, and while an API might come at some point, it doesn't currently exist.

I do seem to recall joeyh talking about supporting further transport protocols(perhaps through hooks). So I'm adding the above links for future reference if this does get implemented.

But IMHO, this doesn't seem like a trivial feature to add. It might have to take some refactoring of some core git-annex parts. Certain things have to be changed quite a bit.

Currently a git-annex client doesn't really require anything(except rsync) to sync from a remote. With bittorrent with DHT support to share between clients, suddenly git-annex will have to maintain a constant bittorrent thread(maybe multiple) that constantly seeds all the files in the git-annex repository, while waiting for a potential remote to request data.

So even if this happens, it is probably gonna take some time.

Just my 2cents.
"""]]


code: Having it named starting with "Add to ..." would improve its convenience visibility (would even position before "Add to Dropbox" ;) )


code:
There are two different possible ways git-annex could use bittorrent:

Let's describe those one by one.

[[!toc]]

Downloading files from multiple git-annex sources simultaneously
================================================================

Having your remotes (optionally!) act like a swarm would be an awesome feature to have because you bring in a lot of new features that optimize storage, bandwidth, and overall traffic usage. This would be made a lot easier if parts of it were implemented in small steps that added a nifty feature. The best part is, each of these could be implemented by themselves, and they're all features that would be really useful.

 1. Concurrent downloads of a file from remotes.

    This would make sense to have, it saves upload traffic on your remotes, and you also get faster DL speeds on the receiving end.

 2. Implementing part of the super-seeding capabilities.

    You upload pieces of a file to different remotes from your laptop, and on your desktop you can download all those pieces and put them together again to get a complete file. If you really wanted to get fancy, you could build in redundancy (ala RAID) so if a remote or two gets lost, you don't lose the entire file. This would be a very efficient use of storage if you have a bunch of free cloud storage accounts (~1GB each) and some big files you want to back up.

 3. Setting it up so that those remotes could talk to one another and share those pieces.

    This is where it gets more like bittorrent. Useful because you upload 1 copy and in a few hours, have say, 5 complete copies on 5 different remotes. You could add or remove remotes from a swarm locally, and push those changes to those remotes, which then adapt themselves to suit the new rules and share those with other remotes in the swarm (rules should be GPG-signed as a safety precaution). Also, if/when deltas get implemented, you could push that delta to the swarm and have all the remotes adopt it. This is cooler than regular bittorrent because the shared file can be updated. As a safety precaution, the delta could be GPG signed so a corrupt file doesn't contaminate the entire swarm. Each remote could have bandwidth/storage limits set in a dotfile.

This is a high-level idea of how it might work, and it's also a HUGE set of features to add, but if implemented, you'd be saving a ton of resources, adding new use cases, and making git-annex more flexible.

Obviously, Step 3 would only work on remotes that you have control of processes on, but if given login credentials to cloud storage remotes (potentially dangerous!) they could read/write to something like dropbox or rsync.

Another thing, this would be completely trackerless. You just use remote groups (or create swarm definitions) and share those with your remotes. **It's completely decentralized!**

This was originally posted [[as a forum post|forum/Wishlist:_Bittorrent-like_transfers]] by [[users/GLITTAH]].

Update: note how [[design/assistant/telehash/]] may be able to answer this specific use case.

Using an external client (addurl torrent support)
=================================================

The alternative to this would be to add `addurl` support for bittorrent files. The same way we can now add Youtube videos to a git-annex repository thanks to [[quvi]], we could also simply do:

    git annex addtorrent debian-live-7.0.0-amd64-standard.iso.torrent

or even better:

    git annex addurl http://cdimage.debian.org/debian-cd/current-live/amd64/bt-hybrid/debian-live-7.0.0-amd64-standard.iso.torrent

This way, a torrent would just become another source for a specific file. When we `get` the file, it fires up `$YOUR_FAVORITE_TORRENT_CLIENT` to download the file.

That way we avoid the implementation complexity of shoving a complete bittorrent client within the assistant. The `get` operation would block until the torrent is downloaded, i guess... --[[anarcat]]


code:
[[!comment format=mdwn
 username="Gastlag"
 ip="109.190.97.30"
 subject="Gittorrent"
 date="2013-08-28T21:49:56Z"
 content="""
May this could interest you : few years ago somes tried to mix Git and Bittorrent.

http://www.advogato.org/article/994.html
http://utsl.gen.nz/gittorrent/rfc.html
http://code.google.com/p/gittorrent/
https://git.wiki.kernel.org/index.php/SoC2010Application#Did_your_organization_participate_in_past_GSoCs.3F_If_so.2C_please_summarize_your_involvement_and_the_successes_and_challenges_of_your_participation
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmTIL7ubr5opWM69Q5VtCxuxC2H0SSnzic"
 nickname="Petr"
 subject="Regarding the approach to connect all nodes in one network and stream from them"
 date="2013-12-29T07:32:05Z"
 content="""
Have a look at http://socialvpn.wordpress.com/. Behind that project is couple of interesting tools worth to investigate.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 3"
 date="2014-01-02T01:31:05Z"
 content="""
> git annex addurl http://cdimage.debian.org/debian-cd/current-live/amd64/bt-hybrid/debian-live-7.0.0-amd64-standard.iso.torrent

This is bad idea; it means you will be reliant on torrent files and trackers when magnet links are the way to go.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 4"
 date="2014-01-02T01:32:37Z"
 content="""
There's a bittorrent library for Haskell, btw: http://hackage.haskell.org/package/bittorrent

Native support in form of a special remote would arguably be best...
"""]]


code:
[[!comment format=mdwn
 username="https://id.koumbit.net/anarcat"
 ip="2001:1928:1:9::1"
 subject="comment 5"
 date="2014-04-01T04:43:16Z"
 content="""
re #3, sure, magnet link support would be awesome as well but i'd prefer to start with something i could digest more easily.

looking at the source, it seems to me that the [quvi implementation](http://source.git-annex.branchable.com/?p=source.git;a=commitdiff;h=46b6d75) could serve as an example as to how this would work. more particularly, there's this concept of a [downloader](http://source.git-annex.branchable.com/?p=source.git;a=commitdiff;h=46b6d75#patch5) that can be used to tap into `addurl` directly. there's a check to see if the downloader is supported, for example.

so we would need:

1. see if the URL / magnet link can be turned into a .torrent somehow
2. figure out what the filename(s!) will be
3. start the torrent and wait for its completion, ideally with some progress bar

i asked around to see if transmission-remote could do this, because it would be nice if we could use an existing daemon (instead of having to rebootstrap the whole DHT at every download). so far, I can't see how it could be done cleanly - maybe we would need to use the simpler \"bittorrent\" commandline client, or maybe tap into libtorrent...

in any case, one of the key problems here is that addurl assumes that the URL maps to a single file, not a directory full of file, which is the way bittorrent works. I am not sure how to fix that assumption.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmHp5oVW8Z9v_vHs5FRtlYj80TYMQWVTS0"
 nickname="dhead"
 subject="comment 6"
 date="2014-06-10T02:18:43Z"
 content="""
What about Syncthing protocol ?
https://github.com/calmh/syncthing/blob/master/protocol/PROTOCOL.md
"""]]


code:
Note: This may not be a bug maybe the software doesn't work the way I think.

What steps will reproduce the problem?

Using the webapp, I created a normal repo on my desktop. I then added a 'Removable drive' repo on my usb stick.

I want all my files to be synced and accessibles on both repos so I set the 'Removable drive' repo to 'client'.

What is the expected output? What do you see instead?

When I look in the 'annex' folder on my usb stick. I see a git repo (annex, branches, hooks) instead of seeing the files in the same way the 'annex' repo on my desktop is.

What version of git-annex are you using? On what operating system?

I'm using 9e57edff287ac53fc4b1cefef7271e9ed17f2285 (Fri Feb 22 15:19:25 2013 +0000).

Ubuntu 12.10 x86_64

[[!tag /design/assistant]]
[[!meta title="assistant should set up non-bare repos on removable drives, and update them when syncing with them"]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.253.75"
 subject="comment 1"
 date="2013-02-22T21:46:59Z"
 content="""
You have a bare git repository on your USB stick. The assistant uses bare repositories on USB sticks currently, although recent changes to support FAT make it possible to use non-bare repositories.

For this to really work the way you want it to, the assistant would need to start updating local non-bare repositories whenever it pushed changes to them. Currently the assistant only updates the repository it's running in, so a non-bare repository on USB would lag behind and show an old version of the directory until manually updated with `git annex sync`.
"""]]


code:
[[!comment format=mdwn
 username="Xyem"
 ip="87.194.19.134"
 subject="comment 2"
 date="2013-02-27T00:02:01Z"
 content="""
Struggling to get the assistant to behave properly, including a similar situation to the above.

I start the assistant with 'git annex webapp' and create an annex from the webapp.
Adding an SSH remote (even if just a different directory on localhost) with \"Remote server\", creates the remote directory with what looks like what the contents of .git should be (annex, branches, hooks, objects etc.), regardless of the group chosen.

Judging from your above comment, this means it is creating a bare repository. Why would it be doing this for an SSH remote, where git-annex-shell is available?

System: Arch Linux
git-annex version: 3.20130216
Installed with: yaourt -S git-annex-bin
"""]]


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="173.162.44.162"
 subject="comment 3"
 date="2013-02-27T01:12:29Z"
 content="""
Xyem, if you use git to clone a repo, instead of letting the assistant do it for you, you can get a non-bare repo.  See the walkthrough for some examples.  You can keep its working tree up to date with \"git annex sync\" run in that directory.

Only helpful if you don't mind using git annex at the command line, I know.

"""]]


code:
[[!comment format=mdwn
 username="Xyem"
 ip="87.194.19.134"
 subject="comment 4"
 date="2013-02-27T12:16:45Z"
 content="""
@edheil:
Yes, I have found that creating the repositories outside of the assistant (or directly on each machine through the assistant) and adding the remotes manually doesn't have this issue.. but as I am not setting this up for myself, so dropping to the command line is an issue (she's not averse to it, but also isn't a power user).

It seems that the assistant/webapp is replacing dropbox functionality, while ignoring the functionality of git-annex. Perhaps I am misunderstanding the goal of it..?

My intended use case in the situation where this issue comes up is:

Repository on desktop has a copy of all files in indirect mode.
Repository on netbook only has a copy of files added locally or fetched manually.
Dropping a file in the netbook annex should cause it to be uploaded to the desktop repository (when it is available). Files should be fetchable/droppable (and unlockable) using the webapp from the desktop/netbook repository respectively.

But it looks like this is (currently) unachievable as the webapp provides no method of fetching/dropping files, the assistant does not create the remote repositories correctly and requires dropping to the command line if worked around. Eep!

Hope this doesn't come off as sour complaining. I was introduced to git-annex by a friend about a week ago (she only learned about it about 2-3 weeks ago and *loves* it) and it is a very nice and flexible tool which fits my own use case perfectly, meeting and exceeding every expectation.. until it came to the assistant and webapp (which makes me wonder why Joey is working on an Android port..).
"""]]


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="173.162.44.162"
 subject="comment 5"
 date="2013-02-27T17:05:07Z"
 content="""
There is a way to fetch/drop files if you don't want to use the command line, though it's not nearly as flexible as using a command line.  If you've got a machine set as \"client\" then dragging files into a subdirectory named \"archive\" drops them, and dragging them out of that subdirectory fetches them.

Honestly, moving files in and out of directories, and preferred content settings, have both been subject to the occasional regression, so there have been times this hasn't worked right, but in theory, that's how you do it.

Unfortunately if you've got more than one \"client\" machine, this means adding/dropping files on one will affect them all, as file movements propagate.
"""]]


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="173.162.44.162"
 subject="comment 6"
 date="2013-02-27T17:06:45Z"
 content="""
also... I've never used the feature where you have two local machines running git-annex and you tell one to share a repo with the other through the web interface, but maybe it would be helpful for this scenario?  Apologies if you've already tried that.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 7"
 date="2013-02-27T17:43:45Z"
 content="""
The way this is supposed to (and does) work is that you use the webapp to create a repository on each computer, and then you pair them together using either local pairing or xmpp pairing.

Today's release of git-annex also adds UI in the webapp to create additional local repositories that are connected to the current repository. However these are not really intended to be put on removable drives since the assistant needs to be running on them to keep them up-to-date.
"""]]


code:
[[!comment format=mdwn
 username="Xyem"
 ip="87.194.19.134"
 subject="comment 8"
 date="2013-02-27T18:36:16Z"
 content="""
@Joey:
As my use case is 1 person using multiple machines, I didn't think local/XMPP pairing was appropriate.

Local pairing because the netbook leaves the network the desktop machine is on, which is when transferring the files is wanted (otherwise, they would just use the desktop...).
XMPP pairing description implies that it is for 2 different people (thus 2 different addresses). Can it be used where both annexes are using the same address? Or does the address also allow you to include the location (i.e. name@host.com/netbook)?

It definitely sounds like I am misunderstanding how this is supposed to work so I apologise for that. Please keep up the excellent work. git-annex is one of those tools I wish I had learned about a long time ago!

@edheil: Interesting workaround, thanks.
"""]]


code:
[[!comment format=mdwn
 username="https://id.koumbit.net/anarcat"
 ip="72.0.72.144"
 subject="confusing..."
 date="2014-05-14T05:04:01Z"
 content="""
this is still really confusing. having to create the repo by hand is a really confusing step - i have setup an external drive to have my files with me as i move around, having to go through hoops to find them in a bare repository seems contrary to the spirit of \"future proofing\" inherent to git annex.

the solution i found was to `git init` the repo on the removable drive before adding it through the webapp. i hope this can eventually be fixed, as it led to confusion among many other users (e.g. [[forum/Accessing_files_in_bare_repository/]], [[bugs/Adding a repository as a \"remote server\" creates a bare repository next to the existing one/]],  [[forum/remote server client repositories are bare!?/]] and so on). --[[anarcat]]
"""]]


code:
The special remote `directory` support files split in chunks, other special remotes do not.

Support for chunks is useful, for example, to be able to upload large files over slow, unreliable connections or to minimize the amount of data to be sent when only part of a big file has been changed.

Couldn't the code used to split, checksum and reconstruct the files in the `directory` remote be used also in all the other special remotes?


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 1"
 date="2014-07-11T20:17:05Z"
 content="""
See [[design/assistant/chunks]]
"""]]


code:
I've just come across this issue and I'm not sure if git-annex is the right place to put it, but in case it is easy enough to do.. may as well ask!

In this scenario, an online service (Bandcamp), automatically creates the archive file when downloading an album each and every time you download it. This results in identical files inside a zip, but different hashes due to the slightly different timestamps on the archive itself.

Would it be possible for git-annex to be able to detect this scenario (in a manner similar to zipcmp) and redirect an add/import to the already existing copy?

I've found this due to trying to decommission an old annex by `git annex import --clean-duplicates ~/annex_old/.git/annex/objects` and finding these files being left.


code:
I periodically move unused files to one of my servers. What I would like to
do is drop any unused file that has been unused for say more than 6 months?
I would like to not drop all unused files.

> It strikes me that this is quite similar to how git handles deleting
> stale refs with the reflog. So, if `git annex unused` were changed to
> also look at the reflog, it would keep all files referred to by all refs
> in the reflog, until the reflog expires. You could then set reflog expiry
> to 6 months, and be done.
>
> However, I think that many users expect git annex unused to be able to
> immediately find and remove a file after it's been deleted. So this
> probably needs to be a configurable behavior. --[[Joey]]


code:
The automatic conflict resolution currently seems to only work within
the context of sync, when merging «synced/foo» into «foo». It would be
convenient if this functionality were exposed for manual merges
between local branches.

E.g., one might invoke «git annex merge» or «git annex autoresolve»
after «git merge» when conflicts are found.

> [[done]] as resolvemerge. --[[Joey]]


code:
I’m just experimenting with git-annex on a repo of ~150k files in about the same number of directories (WORM-backend). Calling, e.g., «git annex status» will take several minutes while stat()-ing all the files (for changes, I presume).

This might already be on you todo list, but I was wondering whether it is possible to increase the performance of «annex status» (or related commands) when «annex watch» is running in the background. In that case, «status» could rely on cached data built-up at some point during initialization, plus based on the data that was accumulated via inotify. (Hopefully, all this won’t even be needed anymore on btrfs at some point in the future.)

(I’m not very knowledgable in these things, so just out of curiosity: I noticed that, even though the «status» invocation takes ages, no HDD activity occurs and all the metadata is probably already in the Linux caches from a run I conducted immediately beforehand. Why do you figure that is? Is context switching so hugely expensive?)


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 1"
 date="2014-07-10T19:37:38Z"
 content="""
Neither `git annex watch` nor the `assistant` persistently store any data about the files in the repository in memory. They cannot speed up `git annex status`.

I'm not sure what's the point of running `git annex status` while the daemon is running, since the daemon will normally immediately notice changes and commit them. The status output should thus generally be empty.

FWIW, `git annex status` takes 0.3s in my largest repo (55k files), on an SSD. That's in indirect mode, and the time is almost completely spent in running `git ls-files --modified`, which is probably about as fast as it can be. In a direct mode repository, it will be rather slower, since it has to query git for the key that was last committed for each file, in order to look up that key's info and see if the file has been modified.
"""]]


code:
[[!comment format=mdwn
 username="zardoz"
 ip="134.147.14.84"
 subject="comment 2"
 date="2014-07-11T11:46:08Z"
 content="""
Yes, you’re probably right that the benefit of this is slim when the
watching daemon auto-adds new files. So the «status» output would
never change and show the status that upheld before starting the
daemon.

The reason I brought this up that I recall reading a comment of yours
somewhere on the site, to the effect that the assistant can sometimes
speed up certain operations, because it can make certain valid
assumptions on the state of the repo due to the fact that the
assistant has been monitoring it. I don’t recall what those operations
were, though. That’s why it occurred to me whether there might be a
daemon that just monitors via inotify, and neither adds nor syncs, and
only provides information to certain commands to speed things up under
some circumstances.

In general, is it accurate to say that git-annex mostly takes the
«space» option when making a space/time-trade-offs? I noticed that the
memory consumption is really slim most of the time, and wondered
whether there might be options of speeding operations up by relying on
more memory instead (perhaps also doing persistent caching). On the
other hand, in some regards you are probably committed to the
time/memory trade-offs taken by vanilla git, so maybe there’s not much
room for improvement, git-annex wise…

Maybe direct-mode repos on the order of 100k’s of files are just not
practical. I’m using indirect mode for my really big repos now, and
it’s now responsive enough to use (except for «annex unused», which is
inherently expensive, as you once explained). At least commiting won’t
take tens of minutes that way. I’ll just have to make the software
play nicely with the symlinks.

BTW, the file-system seems to have a huge impact on this. My large
direct mode annex is practically unusable on ext (tens of minutes per
commit), but still usable on btrfs (a few minutes). I’m migrating one
disk to btrfs at home and will do some controlled benchmarks then. The
added bonus is that directories don’t always take up a full block.

"""]]


code:
[[!comment format=mdwn
 username="zardoz"
 ip="134.147.14.84"
 subject="comment 3"
 date="2014-07-11T11:54:26Z"
 content="""
Ah, interesting, some caching ideas are already discussed at [https://git-annex.branchable.com/design/caching_database/].
"""]]