code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlRThEwuPnr8_bcuuCTQ0rQd3w6AfeMiLY"
 nickname="Alex"
 subject="'webhook' special remote?"
 date="2013-02-24T15:05:27Z"
 content="""
Is there any chance a special remote that functions like a hybrid of 'web' and 'hook'? At least in theory, it should be relatively simple, since it would only support 'get' and the only meaningful parameters to pass would be the URL and the output file name.

Maybe make it something like git config annex.myprogram-webhook 'myprogram $ANNEX_URL $ANNEX_FILE', and fetching could work by adding a --handler or --type parameter to addurl.

The use case here is anywhere that simple 'fetch the file over HTTP/FTP/etc' isn't workable - maybe it's on rapidshare and you need to use plowshare to download it; maybe it's a youtube video and you want to use youtube-dl, maybe it's a chapter of a manga and you want to turn it into a CBZ file when you fetch it.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlRThEwuPnr8_bcuuCTQ0rQd3w6AfeMiLY"
 nickname="Alex"
 subject="comment 9"
 date="2013-02-24T15:13:16Z"
 content="""
A *ridiculously* cool possibility would be to allow them to match against URLs and then handle those (youtube-dl for youtube video URLs, for instance), but that would be additional work on your end and isn't really necessary.
"""]]

repo_name: kakafarm/git

license: unknown


code:
This special remote type stores file contents in a
[ddar](https://github.com/basak/ddar) repository. This provides easy
de-duplication when you use git-annex to manage many files that are similar.

Unlike bup, ddar uses its own storage format, which allows for both creation
and deletion of de-deduplicated files. In addition to using local storage, ddar
archives can be remote, providing that ddar is installed on the remote machine
and ssh is available to it.

See [[walkthrough/using_ddar]] for usage examples.

## encryption

Encryption is nominally supported, but is not useful. Since effective
encryption necessarily obfuscates file contents, similar areas across different
files are no longer visible to ddar and cannot be de-duplicated.

## compression

The same caveat with encryption also generally applies to compression, since
file compression changes file contents such that similar regions across files
no longer appear similar. An exception is `gzip --rsyncable`, which is
specifically designed to work around this issue. This is the only compression
mechanism with which de-duplication remains effective.

## configuration

These parameters can be passed to `git annex initremote` to configure ddar:

* `encryption` - One of "none", "hybrid", "shared", or "pubkey".
  See [[encryption]]. However, note that encryption renders all de-duplication
ineffective.

* `keyid` - Specifies the gpg key to use for [[encryption]].

* `ddarrepo` - Required. This is passed to `ddar` as the path to the ddar
  archive to use. If it doesn't exist, the ddar repository will be created
  automatically when a file is first copied to it. To use a remote ddar
  repository, use a colon (`:`) to separate the hostname from the remote path.
  Example: "ddarrepo=example.com:/big/myddar" or "ddarrepo=/big/myddar"


code:
This special remote type stores file contents in directory.

One use case for this would be if you have a removable drive that
you want to use to sneakernet files between systems (possibly with
[[encrypted|encryption]] contents). Just set up both systems to use
the drive's mountpoint as a directory remote.

Note that directory remotes have a special directory structure
(by design, the same as the \[[rsync|rsync]] remote).
If you just want two copies of your repository with the files "visible"
in the tree in both, the directory special remote is not what you want.
Instead, you should use a regular `git clone` of your git-annex repository.

## configuration

These parameters can be passed to `git annex initremote` to configure the
remote:

* `directory` - The path to the directory where the files should be stored
  for the remote. The directory must already exist. Typically this will
  be an empty directory, or a directory already used as a directory remote.

* `encryption` - One of "none", "hybrid", "shared", or "pubkey".
  See [[encryption]].

* `keyid` - Specifies the gpg key to use for [[encryption]].

* `chunk` - Enables [[chunking]] when storing large files.

* `chunksize` - Deprecated version of chunk parameter above.
  Do not use for new remotes. It is not safe to change the chunksize
  setting of an existing remote.

Setup example:

	# git annex initremote usbdrive type=directory directory=/media/usbdrive/ encryption=none
	# git annex describe usbdrive "usb drive on /media/usbdrive/"


code:
[[!comment format=mdwn
 username="dietz"
 ip="128.119.40.196"
 subject="annexing external files"
 date="2013-07-18T20:57:53Z"
 content="""
This is great work. I've developed a serious annex-addiction and now I want to use it everywhere! In particular I was hoping to apply it to this use case:

I have large files/directories (approx 5 TB) on an nfs mount to which is a) write-protected (think \"read-only medium\") and b) used by non-git users. Both reasons prevent me from setting up a git-annex repos there. However, I would like to use git-annex to keep track of the paths and get/drop files from my different computers.

On one of my servers, I set up a git annex repos, hoping to only manage the structure, the locations, and the number of copies. I don't want to have copies of the 5TB files in that repository, as disk space is not unlimited (just for the sake of making them available to my laptop).

I as banking on using a special remote (either directory or rsync) to tell the git-annex repos where the actual data is.

I am not concerned with data loss, as it is backed up in regular time intervals by our sysadmin.

I tried both directory remote and rsync remote but there seems to be a missing piece (I suppose its add). Any ideas?

This is what I did:

I added the directory remote and an rsync remote

- $ git annex initremote collections type=directory directory=/my/nfs/dir encryption=none
- $ git annex initremote rsync type=rsync rsyncurl=ssh://myserver/mnt/nfs/dir encryption=none

the copy command fails without complaints
 $ git annex copy --from collections

I tried adding virtual files to git annex

- $ git annex add file/inside/dir

but still any kind of get/copy command does not get any new files

It would be awesome if I could use git-annex for this, to keep track of my copies and copies of copies. And then I could also keep track of data on my write-protected DVDs.

Is there any chance?

Thanks a lot!

-- Laura

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.2.25"
 subject="comment 2"
 date="2013-07-19T13:54:10Z"
 content="""
@Laura the directory special remote requires files to
be in a particular directory structure with special names
git-annex comes up with. So you can't use it on an existing
tree of files like that.

What you can do is use the [[web_special_remote|web]],
with a `file://` url to point to the files wherever
they are stored. So for example,
`git annex addurl file:///media/dvd/file`
"""]]


code:
[[!comment format=mdwn
 username="dietz"
 ip="128.119.40.196"
 subject="comment 12"
 date="2013-07-20T06:06:31Z"
 content="""
Using the web remote is a pretty nice trick!

Thanks, Joey - I would not have guessed that.

-- Laura
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlc1og3PqIGudOMkFNrCCNg66vB7s-jLpc"
 nickname="Paul"
 subject="how is this different than rsync?"
 date="2012-06-22T22:10:19Z"
 content="""
Thanks for this great tool! I was wondering what the differences are between using `type=directory`, `type=rsync`, or a bare git repo for directories?

I guess I can't use just a regular repo because my USB drive is formatted as `vfat` -- which threw me for a loop the first time I heard about `git-annex` about a year ago, because I followed the walkthrough, and it didn't work as expected and gave up (now I know it was just a case of PEBKAC). It might be worth adding a note about [vfat](http://git-annex.branchable.com/bugs/fat_support/) to the \"Adding a remote\" section of the [walkthrough](http://git-annex.branchable.com/walkthrough/), since the unstated assumption there is that the USB drive is formatted as a filesystem that supports symlinks.

Thanks again, my scientific data management just got a lot more sane!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.2.25"
 subject="comment 2"
 date="2012-06-25T15:29:29Z"
 content="""
The directory and rsync special remotes intentionally use the same layout. So the same directory could be set up as both types of special remotes.

The main reason to use this rather than a bare git repo is that it supports encryption.
"""]]


code:
[[!comment format=mdwn
 username="https://dzsino.myopenid.com/"
 nickname="dzsino"
 subject="dropping a directory remote?"
 date="2013-01-15T22:29:15Z"
 content="""
How do I drop a directory remote after initremote? Say I want to start over and enable chunking (not supposed to be enabled on an existing directory), can I simply git remote rm it?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.7.238"
 subject="comment 4"
 date="2013-01-17T18:22:26Z"
 content="""
The best way to remove a special remote is to first `git annex move --from $remote` to get all the content out of it, then `git annex dead $remote` and finally you can `git remote rm $remote`
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawk3HGoDpnOPob5jOjvIootmkve1-nCpRiI"
 nickname="Kalle"
 subject="Directory remote with files accessible from non git-annex system?"
 date="2013-01-20T15:44:16Z"
 content="""

"""]]


code:
[[!comment format=mdwn
 username="http://nicolas-schodet.myopenid.com/"
 ip="2a01:e35:8ae6:f130:1e4b:d6ff:fe78:1ddb"
 subject="comment 6"
 date="2013-06-26T17:52:12Z"
 content="""
I tried the suggestion on comment 4, but when I add again a remote with the same path, it gets the same repository identifier and is considered dead. Is that expected?

My use case: I use a usb drive to transfer some large files from one git annex to another, then I use the usb drive for something else and the special remote is removed.  Later I want to use the same usb drive again, but when I create the repository, it starts in the dead state.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.193"
 subject="comment 7"
 date="2013-06-26T18:30:39Z"
 content="""
@nicolas, I suspect you are using `git annex initremote` with the same name that you used for the now dead-and-buried remote. That causes it to be reanimated, which is not what you want.

Since git-annex version 4.20130501, `git annex initremote` is reserved for creating new remotes, not enabling old ones, so it will refuse to do this. That's to avoid exactly this confusion.

Using `git annex initremote` with a different remote name, and the same directory should work just fine.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmRFKwny4rArBaz-36xTcsJYqKIgdDaw5Q"
 nickname="Andrew"
 subject="comment 8"
 date="2013-06-26T18:46:40Z"
 content="""
For the use case you're describing, it might be better to define the usb key as a remote set to \"manual.\" Then, you can copy over the things you want with git annex copy --to=usbkey and when you're done drop everything with git annex drop --from=usbkey and never destroy the remote.
"""]]


code:
[[!comment format=mdwn
 username="http://nicolas-schodet.myopenid.com/"
 ip="2a01:e35:8ae6:f130:1e4b:d6ff:fe78:1ddb"
 subject="comment 9"
 date="2013-06-26T20:05:20Z"
 content="""
Thanks for your answers. You're right, the simplest solution for me is to never remove the remote. If my directory is lost, I realized that I can simply create an empty directory.
"""]]


code:
There are three ways to implement a new special remote:

1. Using the [[hook]] special remote to tell git-annex what commands
   to run to store and retrieve data. This is the easiest way, and
   is great for prototyping.
2. Writing it in Haskell and adding it to git-annex.
3. Writing a program in any language you like that speaks the
   [[design/external_special_remote_protocol]].

The rest of this page concentrates on writing new external special remotes.
It's not hard!

* All you need is to make a program with a name like `git-annex-remote-$bar`.
* Install it in PATH.
* When the user runs `git annex initremote foo type=external externaltype=$bar`,
  it will use your program.
* If things don't seem to work, pass `--debug` and you'll see, amoung other
  things, a transcript of git-annex's communication with your program.
* If you build a new special remote, please add it to the list
  of [[special_remotes]].

Here's a simple shell script example, which can easily be adapted
to run whatever commands you need. Or better, re-written in some better
language of your choice. See [[design/external_special_remote_protocol]]
for the details.

[[!inline pages="special_remotes/external/example.sh" feeds=no]]


code:
#!/bin/sh
# git-annex external special remote program
#
# This is basically the same as git-annex's built-in directory special remote.
#
# Install in PATH as git-annex-remote-directory
#
# Copyright 2013 Joey Hess; licenced under the GNU GPL version 3 or higher.

set -e

# This program speaks a line-based protocol on stdin and stdout.
# When running any commands, their stdout should be redirected to stderr
# (or /dev/null) to avoid messing up the protocol.
runcmd () {
	"$@" >&2
}

# Gets a value from the remote's configuration, and stores it in RET
getconfig () {
	ask GETCONFIG "$1"
}

# Stores a value in the remote's configuration.
setconfig () {
	echo SETCONFIG "$1" "$2"
}

# Sets LOC to the location to use to store a key.
calclocation () {
	ask DIRHASH "$1"
	LOC="$mydirectory/$RET/$1"
}

# Asks for some value, and stores it in RET
ask () {
	echo "$1" "$2"
	read resp
	# Tricky POSIX shell code to split first word of the resp,
	# preserving all other whitespace
	case "${resp%% *}" in
		VALUE)
			RET="$(echo "$resp" | sed 's/^VALUE \?//')"
		;;
		*)
			RET=""
		;;
	esac
}

# This remote doesn't need credentials to access it,
# but many of them will. Here's how to handle requiring the user
# set MYPASSWORD and MYLOGIN when running initremote. The creds
# will be stored securely for later use, so the user only needs
# to provide them once.
setupcreds () {
	if [ -z "$MYPASSWORD" ] || [ -z "$MYLOGIN" ]; then
		echo INITREMOTE-FAILURE "You need to set MYPASSWORD and MYLOGIN environment variables when running initremote."
	else
		echo SETCREDS mycreds "$MYLOGIN" "$MYPASSWORD"
		echo INITREMOTE-SUCCESS
	fi
}

getcreds () {
	echo GETCREDS mycreds
	read resp
	case "${resp%% *}" in
		CREDS)
			MYLOGIN="$(echo "$resp" | sed 's/^CREDS \([^ ]*\) .*/\1/')"
			MYPASSWORD="$(echo "$resp" | sed 's/^CREDS [^ ]* //')"
		;;
	esac

}

# This has to come first, to get the protocol started.
echo VERSION 1

while read line; do
	set -- $line
	case "$1" in
		INITREMOTE)
			# Do anything necessary to create resources
			# used by the remote. Try to be idempotent.
			#
			# Use GETCONFIG to get any needed configuration
			# settings, and SETCONFIG to set any persistent
			# configuration settings.
			#
			# (Note that this is not run every time, only when
			# git annex initremote or git annex enableremote is
			# run.)

			# The directory provided by the user
			# could be relative; make it absolute,
			# and store that.
			getconfig directory
			mydirectory="$(readlink -f "$RET")" || true
			setconfig directory "$mydirectory"
			if [ -z "$mydirectory" ]; then
				echo INITREMOTE-FAILURE "You need to set directory="
			else
				if mkdir -p "$mydirectory"; then
					setupcreds
				else
					echo INITREMOTE-FAILURE "Failed to write to $mydirectory"
				fi
			fi
		;;
		PREPARE)
			# Use GETCONFIG to get configuration settings,
			# and do anything needed to get ready for using the
			# special remote here.
			getcreds
			getconfig directory
			mydirectory="$RET"
			if [ -d "$mydirectory" ]; then
				echo PREPARE-SUCCESS
			else
				echo PREPARE-FAILURE "$mydirectory not found"
			fi
		;;
		TRANSFER)
			key="$3"
			file="$4"
			case "$2" in
				STORE)
					# Store the file to a location
					# based on the key.
					# XXX when at all possible, send PROGRESS
					calclocation "$key"
					mkdir -p "$(dirname "$LOC")"
					# Store in temp file first, so that
					# CHECKPRESENT does not see it
					# until it is all stored.
					mkdir -p "$mydirectory/tmp"
					tmp="$mydirectory/tmp/$key"
					if runcmd cp "$file" "$tmp" \
					   && runcmd mv -f "$tmp" "$LOC"; then
						echo TRANSFER-SUCCESS STORE "$key"
					else
						echo TRANSFER-FAILURE STORE "$key"
					fi

					mkdir -p "$(dirname "$LOC")"
					# The file may already exist, so
					# make sure we can overwrite it.
					chmod 644 "$LOC" 2>/dev/null || true
				;;
				RETRIEVE)
					# Retrieve from a location based on
					# the key, outputting to the file.
					# XXX when easy to do, send PROGRESS
					calclocation "$key"
					if runcmd cp "$LOC" "$file"; then
						echo TRANSFER-SUCCESS RETRIEVE "$key"
					else
						echo TRANSFER-FAILURE RETRIEVE "$key"
					fi
				;;
			esac
		;;
		CHECKPRESENT)
			key="$2"
			calclocation "$key"
			if [ -e "$LOC" ]; then
				echo CHECKPRESENT-SUCCESS "$key"
			else
				if [ -d "$mydirectory" ]; then
					echo CHECKPRESENT-FAILURE "$key"
				else
					# When the directory does not exist,
					# the remote is not available.
					# (A network remote would similarly
					# fail with CHECKPRESENT-UNKNOWN
					# if it couldn't be contacted).
					echo CHECKPRESENT-UNKNOWN "$key" "this remote is not currently available"
				fi
			fi
		;;
		REMOVE)
			key="$2"
			calclocation "$key"
			# Note that it's not a failure to remove a
			# key that is not present.
			if [ -e "$LOC" ]; then
				if runcmd rm -f "$LOC"; then
					echo REMOVE-SUCCESS "$key"
				else
					echo REMOVE-FAILURE "$key"
				fi
			else
				echo REMOVE-SUCCESS "$key"
			fi
		;;
		*)
			# The requests listed above are all the ones
			# that are required to be supported, so it's fine
			# to say that any other request is unsupported.
			echo UNSUPPORTED-REQUEST
		;;
	esac
done

# XXX anything that needs to be done at shutdown can be done here


code:
[git-remote-gcrypt](https://github.com/joeyh/git-remote-gcrypt/)
adds support for encrypted remotes to git. The git-annex gcrypt special
remote allows git-annex to also store its files in such repositories.
Naturally, git-annex encrypts the files it stores too, so everything
stored on the remote is encrypted.

See [[tips/fully_encrypted_git_repositories_with_gcrypt]] for some examples
of using gcrypt.

## configuration

These parameters can be passed to `git annex initremote` to configure
gcrypt:

* `encryption` - One of "none", "hybrid", "shared", or "pubkey".
  See [[encryption]].

* `keyid` - Specifies the gpg key to use for encryption of both the files
  git-annex stores in the repository, as well as to encrypt the git
  repository itself. May be repeated when multiple participants
  should have access to the repository.

* `gitrepo` - Required. The path or url to the git repository
  for gcrypt to use. This repository should be either empty, or an existing
  gcrypt repositry.

* `shellescape` - See [[rsync]] for the details of this option.

## notes

For git-annex to store files in a repository on a remote server, you need
shell access, and `rsync` must be installed. Those are the minimum
requirements, but it's also recommended to install git-annex on the remote
server, so that [[git-annex-shell]] can be used.

While you can use git-remote-gcrypt with servers like github, git-annex
can't store files on them. In such a case, you can just use
git-remote-gcrypt directly.

If you use encryption=hybrid, you can add more gpg keys that can access
the files git-annex stored in the gcrypt repository. However, due to the
way git-remote-gcrypt encrypts the git repository, you will need to somehow
force it to re-push everything again, so that the encrypted repository can
be decrypted by the added keys. Probably this can be done by setting
`GCRYPT_FULL_REPACK` and doing a forced push of branches.

Recent versions of git-annex configure gcrypt-publish-participants when
setting up a gcrypt repository. This is done to avoid unncessary gpg
passphrase prompts, but it does publish the gpg keyids that can decrypt the
repository. Unset it if you need to obscure that.


code:
This special remote type stores file contents in Amazon Glacier.

To use it, you need to have [glacier-cli](http://github.com/basak/glacier-cli)
installed.

The unusual thing about Amazon Glacier is the multiple-hour delay it takes
to retrieve information out of Glacier. To deal with this, commands like
"git-annex get" request Glacier start the retrieval process, and will fail
due to the data not yet being available. You can then wait approximately
four hours, re-run the same command, and this time, it will actually
download the data.

## configuration

The standard environment variables `AWS_ACCESS_KEY_ID` and
`AWS_SECRET_ACCESS_KEY` are used to supply login credentials
for Amazon. You need to set these only when running
`git annex initremote`, as they will be cached in a file only you
can read inside the local git repository.

A number of parameters can be passed to `git annex initremote` to configure
the Glacier remote.

* `encryption` - One of "none", "hybrid", "shared", or "pubkey".
  See [[encryption]].

* `keyid` - Specifies the gpg key to use for [[encryption]].

* `embedcreds` - Optional. Set to "yes" embed the login credentials inside
  the git repository, which allows other clones to also access them. This is
  the default when gpg encryption is enabled; the credentials are stored
  encrypted and only those with the repository's keys can access them.

  It is not the default when using shared encryption, or no encryption.
  Think carefully about who can access your repository before using
  embedcreds without gpg encryption.

* `datacenter` - Defaults to "us-east-1".

* `vault` - By default, a vault name is chosen based on the remote name
  and UUID. This can be specified to pick a vault name.

* `fileprefix` - By default, git-annex places files in a tree rooted at the
  top of the Glacier vault. When this is set, it's prefixed to the filenames
  used. For example, you could set it to "foo/" in one special remote,
  and to "bar/" in another special remote, and both special remotes could
  then use the same vault.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmUJBh1lYmvfCCiGr3yrdx-QhuLCSRnU5c"
 nickname="Justin"
 subject="comment 1"
 date="2013-05-16T00:54:21Z"
 content="""
The glacier-cli tool seems to have been abandoned, and there are a number of outstanding issues with it.  boto has a `glacier` tool, but it doesn't seem to include caching, which seems to be something git annex needs.

Looking through the PRs, it seems like we should build a tool specifically tailored to git annex's needs.  It seems that there are at least three of us willing to hack on this if it's in Python.  I'm not sure any of us knows haskell, though...
"""]]


code:
[[!comment format=mdwn
 username="basak"
 ip="2001:8b0:1c8::2"
 subject="comment 2"
 date="2013-05-17T08:35:10Z"
 content="""
I'm the glacier-cli author. It is not abandoned!

glacier-cli is supposed to map to Glacier exactly, so that it is compatible with all other tools. Most of the outstanding PRs break this essential behaviour, so I have not merged them. Many of the feature requests and bugs related to the upstream boto library, which is just about the best maintained client library that exists for AWS on any platform (and Amazon have adopted it now, IIRC). I have written appropriate reviews on all the PRs.

If there is specific behaviour that git-annex needs, them I am happy to accept PRs for this, provided that they do not break the ability (and default) for glacier-cli to talk to Glacier natively without an extra layer of interpretation. If an extra layer of interpretation is needed (eg. forbidding duplicate \"keys\"), then this needs to be an option, or wrapped in a separate tool, or written into git-annex's Glacier special remote.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmUJBh1lYmvfCCiGr3yrdx-QhuLCSRnU5c"
 nickname="Justin"
 subject="comment 3"
 date="2013-05-19T05:56:45Z"
 content="""
Hi!  :)

The main issue I'm hitting is the \"Multiple rows were found for one()\" error.  I think I get this when git-annex tries to upload the same file twice (which may be a bug in git-annex, which could apply de-duplication earlier), but I think I also get it when trying to upload a file whose upload I've canceled in the past.

I don't quite understand what git-annex needs here, and I totally understand that you're writing a general-purpose tool.  But there does seem to be an issue that git-annex needs fixed one way or another.

I'm happy to try fixing it myself if you can help me understand what's going on (I didn't quite understand your review in the PR), but if I'm the only person in the world using git-annex to back up to glacier, that scares me a little!

    copy foo/bar/baz (checking glacier...) Traceback (most recent call last):
      File \"/home/jlebar/code/glacier-cli/glacier\", line 694, in <module>
        App().main()
      File \"/home/jlebar/code/glacier-cli/glacier\", line 680, in main
        args.func(args)
      File \"/home/jlebar/code/glacier-cli/glacier\", line 579, in archive_checkpresent
        last_seen = self.cache.get_archive_last_seen(args.vault, args.name)
      File \"/home/jlebar/code/glacier-cli/glacier\", line 157, in get_archive_last_seen
        result = self._get_archive_query_by_ref(vault, ref).one()
      File \"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py\", line 2182, in one
        \"Multiple rows were found for one()\")
    sqlalchemy.orm.exc.MultipleResultsFound: Multiple rows were found for one()

"""]]


code:
[[!comment format=mdwn
 username="basak"
 ip="2001:8b0:1c8::2"
 subject="comment 4"
 date="2013-05-22T18:10:32Z"
 content="""
Let's discuss this in a bug. I've created http://git-annex.branchable.com/bugs/Glacier_remote_uploads_duplicates/
"""]]


code:
[[!comment format=mdwn
 username="http://id.clacke.se/"
 nickname="clacke"
 subject="comment 5"
 date="2013-06-03T09:03:57Z"
 content="""
You are not the only one, Justin. I am just getting into git-annex and I am setting up a glacier remote as I write this.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmZgZuUhZlHpd_AbbcixY0QQiutb2I7GWY"
 nickname="Jimmy"
 subject="I know this thread is old but..."
 date="2013-11-18T00:27:50Z"
 content="""
It's working well for me, though I seem to sometimes still hit the duplicate bug listed above.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlq4ClC5EMN1Vq1DpWXAqP5TiDnCK1mSfk"
 nickname="Jonas"
 subject="Expected cost for a repository"
 date="2013-11-24T22:32:16Z"
 content="""
Is there a way to estimate the cost of storing a repo on glacier?

I'm especially worried because of the cost of STORE and RETRIEVE requests; there are hundreds of thousands of small files in my annex repo, so that request cost could easily dominate storage cost. Does the glacier remote do anything to minimize the number of objects stored in glacier?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlhaXbLZQ5uQ6fM2kV0n8bj6IwZjx2CLeM"
 nickname="Shane"
 subject="Annex can't find Glacier remote"
 date="2014-06-26T05:50:38Z"
 content="""
I'm setting up git-annex with glacier-cli for the first time.  I have installed git-annex via Yum and glacier-cli according to the instructions on Github.  The `glacier` command is in my path.  I did not set up hooks with Git annex as it appears that using hooks for glacier is no longer required.

Here is my version information for git-annex:

	$ git-annex version
	git-annex version: 3.20120523
	local repository version: 3
	default repository version: 3
	supported repository versions: 3
	upgrade supported from repository versions: 0 1 2

When I attempt to add my Glacier remote, here is what I see:

	$ git annex initremote glacier type=glacier keyid=ABCDEFG
	git-annex: Unknown remote type glacier

Is there something else I need to do in order to correctly install Glacier integration with git-annex?  I'm having trouble finding up-to-date information that describes the installation process.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="that is an ancient version of git-annex, Shane"
 date="2014-07-02T17:11:24Z"
 content="""
The first git-annex release to support glacier without using hooks was 3.20121126.
"""]]


code:
This special remote type lets you store content in a remote of your own
devising, configured via some simple hooks.

It's not recommended to use this remote type when another like [[rsync]]
or [[directory]] will do. If your hooks are not carefully written, data
could be lost.

If you're building a special remote for others to use,
instead consider building an [[external_special_remote|external]].

## example

Here's a simple example that stores content on clay tablets. If you
implement this example in the real world, I'd appreciate a tour
next Apert! :) --[[Joey]]

	# git config annex.cuneiform-store-hook 'tocuneiform < "$ANNEX_FILE" | tablet-writer --implement=stylus --title="$ANNEX_KEY" | tablet-proofreader | librarian --shelve --floor=$ANNEX_HASH_1 --shelf=$ANNEX_HASH_2'
	# git config annex.cuneiform-retrieve-hook 'librarian --get --floor=$ANNEX_HASH_1 --shelf=$ANNEX_HASH_2 --title="$ANNEX_KEY" | tablet-reader --implement=coffee --implement=glasses --force-monastic-dedication | fromcuneiform > "$ANNEX_FILE"'
	# git config annex.cuneiform-remove-hook 'librarian --get --floor=$ANNEX_HASH_1 --shelf=$ANNEX_HASH_2 --title="$ANNEX_KEY" | goon --hit-with-hammer'
	# git config annex.cuneiform-checkpresent-hook 'librarian --find --force-distrust-catalog --floor=$ANNEX_HASH_1 --shelf=$ANNEX_HASH_2 --title="$ANNEX_KEY" --shout-title'
	# git annex initremote library type=hook hooktype=cuneiform encryption=none
	# git annex describe library "the reborn Library of Alexandria (upgrade to bronze plates pending)"

Can you spot the potential data loss bugs in the above simple example?
(Hint: What happens when the `tablet-proofreader` exits nonzero?)

## configuration

These parameters can be passed to `git annex initremote`:

* `hooktype` - Required. This specifies a collection of hooks to use for
  this remote.

* `encryption` - One of "none", "hybrid", "shared", or "pubkey".
  See [[encryption]].

* `keyid` - Specifies the gpg key to use for [[encryption]].

## hooks

Each type of hook remote is specified by a collection of hook commands.
Each hook command is run as a shell command line, and should return nonzero
on failure, and zero on success.

These environment variables are used to communicate with the hook commands:

* `ANNEX_KEY` - name of a key to store, retrieve, remove, or check.
* `ANNEX_FILE` - a file containing the key's content
* `ANNEX_HASH_1` - short stable value, based on the key, can be used for hashing
  into 1024 buckets.
* `ANNEX_HASH_2` - another hash value, can be used for a second level of hashing

The settings to use in git config for the hook commands are as follows:

* `annex.$hooktype-store-hook` - Command run to store a key in the special remote.
  `ANNEX_FILE` contains the content to be stored.

* `annex.$hooktype-retrieve-hook` - Command run to retrieve a key from the special remote.
  `ANNEX_FILE` is a file that the retrieved content should be written to.
  The file may already exist with a partial
  copy of the content (or possibly just garbage), to allow for resuming
  of partial transfers.

* `annex.$hooktype-remove-hook` - Command to remove a key from the special remote.

* `annex.$hooktype-checkpresent-hook` - Command to check if a key is present
  in the special remote. Should output the key name to stdout, on its own line,
  if and only if the key has been actively verified to be present in the
  special remote (caching presence information is a very bad idea);
  all other output to stdout will be ignored.

## combined hook program

This interface is deprecated -- it's better, and not much harder,
to write an [[external_special_remote|external]]!

Rather than setting all of the above hooks, you can write a single
program that handles everything, and set a single hook to make it be used.

	# git config annex.demo-hook /usr/local/bin/annexdemo
	# git annex initremote mydemorepo type=hook hooktype=demo encryption=none

The program just needs to look at the `ANNEX_ACTION` environment variable
to see what it's being asked to do. For example:

[[!format sh """
#!/bin/sh
set -e
case "$ANNEX_ACTION" in
	store)
		demo-upload "$ANNEX_HASH_1/$ANNEX_HASH_2/$ANNEX_KEY" < "$ANNEX_FILE"
	;;
	retrieve)
		demo-download "$ANNEX_HASH_1/$ANNEX_HASH_2/$ANNEX_KEY" > "$ANNEX_FILE"
	;;
	remove)
		demo-nuke "$ANNEX_HASH_1/$ANNEX_HASH_2/$ANNEX_KEY"
	;;
	checkpresent)
		if demo-exists "$ANNEX_HASH_1/$ANNEX_HASH_2/$ANNEX_KEY"; then
			echo "$ANNEX_KEY"
		fi
	;;
	*)
		echo "unknown ANNEX_ACTION: $ANNEX_ACTION" >&2
		exit 1
	;;
esac
"""]]


code:
[[!comment format=mdwn
 username="helmut"
 ip="89.0.176.236"
 subject="Asynchronous hooks?"
 date="2012-10-13T09:46:14Z"
 content="""
Is there a way to use asynchronous remotes? Interaction with git annex would have to
split the part of initiating some action from completing it.

I imagine I could `git annex copy` a file to an asynchronous remote and the command
would almost immediately complete. Later I would learn that the transfer is
completed, so the hook must be able to record that information in the `git-annex`
branch. An additional plumbing command seems required here as well as a way to
indicate that even though the store-hook completed, the file is not transferred.

Similarly `git annex get` would immediately return without actually fetching the
file. This should already be possible by returning non-zero from the retrieve-hook.
Later the hook could use plumbing level commands to actually stick the received file
into the repository.

The remove-hook should need no changes, but the checkpresent-hook would be more like
a trigger without any actual result. The extension of the plumbing required for the
extension to the receive-hook could update the location log. A downside here is that
you never know when a fsck has completed.

My proposal does not include a way to track the completion of actions, but relies on
the hook to always complete them reliably. It is not clear that this is the best road
for asynchronous hooks.

One use case for this would be a remote that is only accessible via uucp. Are there
other use cases? Is the drafted interface useful?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnWvnTWY6LrcPB4BzYEBn5mRTpNhg5EtEg"
 nickname="Bence"
 subject="More environment variables"
 date="2013-07-09T10:28:58Z"
 content="""
Could you please include the original filename+path in the environment variables (_next to ANNEX_KEY & ANNEX_FILE_)? Like ANNEX_FILENAME and ANNEX_PATH.

Having these infos in a hook would help eg. a flickr backend to be more usefull. [Tags](http://www.flickr.com/help/tags/) would contain the ANNEX_KEY and the image title could be the original filename (ANNEX_FILENAME). Also, having directory path (ANNEX_PATH) for the given file, the uploading process could put images into the proper sets/collections. Voila, you have a \"filesystem based\" flickr image gallery (almost like flickrfs).

Other backends, like AmazonS3 having [meta data](http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html) also could benefit from this.

To build the Death Star further, an _annex.$hooktype-**sync**-hook_ would instruct the backend to sync data, eg. place or move images/files in the proper image-sets/directories after they are moved/repositioned in git-annex, but that would be the backend's job, not git-annex's. Maybe the sync-hook would be called when _git annex sync_ is called. This is just an idea.

While writing this, a new hook for sharing came into my mind: _annex.$hooktype-**share**-hook_.
Calling this on a file/directory (_git annex share my_image_to_share.jpg_) would return a publicly shareable (short)url pointing to the file/directory. This would work for web-backends like AmazonS3, flickr, DropBox, Google Drive, ...
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmWg4VvDTer9f49Y3z-R0AH16P4d1ygotA"
 nickname="Tobias"
 subject="comment 3"
 date="2013-07-09T10:54:58Z"
 content="""
I have requested this before. But it doesn't seem to be entirely doable because some items may have multiple equally correct filenames/paths. And some items may have zero filenames/paths.

That said I hope a solution can be found because I really want this feature too. And would implement it in all my hooks.

And for some of the cases i don't really see it as an issue. If you have a public flickr repo with clean(unencrypted) files. It is because you want to access existing files. If an object has no filename/path the hook could/would/should just ignore the file, sure this means no backup of old versions of files, but you can have other backends for those versions.

The bigger issue is with the same file existing multiple places in the annex, which filename/path should be used? And the filename/path can change between sync(if it is deleted from one of the positions). I personally still see this as being entirely doable. The key for downloading would always be the same, so worst case scenario is the image may be duplicated on flickr. Or that the picture only one of the multiple folders it should be in on flickr. Still, i see these issues as being minor, and that usability would increase if this was implemented, even with these caveats.

And there probably is some issues I haven't realized/know about.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnWvnTWY6LrcPB4BzYEBn5mRTpNhg5EtEg"
 nickname="Bence"
 subject="checkpresent success and failure"
 date="2013-07-31T13:06:21Z"
 content="""
What value should be returned in the \"checkpresent-hook\" to signal that the given file does not exist in the given backend?

Should the called hook process return an exit code less or greater then zero? In this case the following is displayed:
>(user error (sh [\"-c\",\"name_of_the_process\"] exited 1)) failed

This tells that the process failed (no internet connection or something that prevents the process from doing its job) and not that result is false, which would mean the file/entry does not exist in the given backend.
If the return code is zero the file is treated as existing file/entry (no matter what I write to stderr).

Also I think, the \"checkpresent\" block misses the ending ;; in the example.

Here is my work-in-progress hook: https://gist.github.com/parhuzamos/31bf4516eea434e0d248
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.21"
 subject="comment 5"
 date="2013-07-31T16:25:55Z"
 content="""
The checkpresent hook should always exit 0 unless there was an exceptional condition (eg, perhaps it cannot check if the file exists one way or the other). Like the documentation for it says, the important thing is what it outputs to stdout, which should contain the key name if it's present, and should not contain the key name if it's not present.

I hope you post to this website about your special remote when you get it fully working!
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnWvnTWY6LrcPB4BzYEBn5mRTpNhg5EtEg"
 nickname="Bence"
 subject="comment 6"
 date="2013-07-31T17:34:50Z"
 content="""
Roger that.

If this is acceptable: [terminal output screenshot](http://i.imgur.com/lsJJYwF.png), than I'm almost done and will publish soon.
(Of course a REST API using client would much be better, but this is just the start.)

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.21"
 subject="comment 7"
 date="2013-07-31T17:42:22Z"
 content="""
If I were you I'd suppress that \"File not found\" error.

Hook special remotes *can* output messages to stderr, and it's also fine to output eg, progress bars to stdout when seding/receving files. But unnecessary cluttery output should be avoided.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnWvnTWY6LrcPB4BzYEBn5mRTpNhg5EtEg"
 nickname="Bence"
 subject="checkpresent again"
 date="2013-08-01T23:18:38Z"
 content="""
In the current [HEAD](https://github.com/joeyh/git-annex/commit/bb74db6ef094324062adcf26a677113ee6fd0e58) the \"checkpresent\" method in [Hook.hs](https://github.com/joeyh/git-annex/blob/master/Remote/Hook.hs#L145) is missing a \"return\" while other hooks have a return value eg. [Directory.hs](https://github.com/joeyh/git-annex/blob/master/Remote/Directory.hs#L241), [Rsync.hs](https://github.com/joeyh/git-annex/blob/master/Remote/Rsync.hs#L272), ...

I noticed that if my *checkpresent* hook does not output anything (to be exact: I commented out all the lines in the block) it behaves strangely. I copied the test file to \"copyrepo\", removed it by hand (so git-annex does not know about the change) and executed a fsck.

    ~/annex5/test1/123 $ git annex fsck --from copyrepo --fast
    fsck somefile.1 (checking copyrepo...) (fixing location log)
      ** Based on the location log, somefile.1
      ** was expected to be present, but its content is missing.
    failed

running the check again

    ~/annex5/test1/123 $ ga fsck --from copyrepo --fast
    fsck somefile.1 (checking copyrepo...) ok

and running the check again without --fast

    ~/annex5/test1/123 $ ga fsck --from copyrepo
    fsck somefile.1 (checking copyrepo...) ok

It thinks, the file is in the repo but it is not.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.145"
 subject="comment 9"
 date="2013-08-01T23:51:48Z"
 content="""
The behavior you show with `fsck --from` is that the first time it's run against the damaged remote it notices the file is not present using the checkpresent hook. It then updates the location log. The subsequent times it's run, it sees that the location log says the file is not present in the remote. It verifies this is the case by calling the checkpresent hook. Since the two data sources agree, and numcopies is still satisfied, it prints \"ok\". There does not seem to be a bug here.

(`return` in Haskell does not do what you would expect to happen in a traditional imperative language. It does not alter control flow, and any function using `return` can be mechanically converted to one that does not use `return`.)
"""]]


code:
This special remote type rsyncs file contents to somewhere else.

Setup example:

	# git annex initremote myrsync type=rsync rsyncurl=rsync://rsync.example.com/myrsync keyid=joey@kitenet.net
	# git annex describe myrsync "rsync server"

Or for using rsync over SSH

	# git annex initremote myrsync type=rsync rsyncurl=ssh.example.com:/myrsync keyid=joey@kitenet.net
	# git annex describe myrsync "rsync server"

## configuration

These parameters can be passed to `git annex initremote` to configure rsync:

* `encryption` - One of "none", "hybrid", "shared", or "pubkey".
  See [[encryption]].

* `keyid` - Specifies the gpg key to use for [[encryption]].

* `rsyncurl` - Required. This is the url or `hostname:/directory` to
  pass to rsync to tell it where to store content.

* `shellescape` - Optional. Set to "no" to avoid shell escaping normally
  done when using rsync over ssh. That escaping is needed with typical
  setups, but not with some hosting providers that do not expose rsynced
  filenames to the shell. You'll know you need this option if `git annex get`
  from the special remote fails with an error message containing a single
  quote (`'`) character. If that happens, you can run enableremote
  setting shellescape=no.

The `annex-rsync-options` git configuration setting can be used to pass
parameters to rsync.

## annex-rsync-transport

You can use the `annex-rsync-transport` git configuration setting to choose
whether we run rsync over ssh or rsh.  This setting is also used to specify
parameters that git annex will pass to ssh/rsh.

ssh is the default transport; if you'd like to run rsync over rsh, modify your
.git/config to include

        annex-rsync-transport = rsh

under the appropriate remote.

To pass parameters to ssh/rsh, include the parameters after "rsh" or
"ssh".  For example, to configure ssh to use the private key at
`/path/to/private/key`, specify

         annex-rsync-transport = ssh -i /path/to/private/key

Note that environment variables aren't expanded here, so for example, you
cannot specify `-i $HOME/.ssh/private_key`.


code:
[[!comment format=mdwn
 username="http://cstork.org/"
 nickname="Chris Stork"
 subject="comment 10"
 date="2013-08-25T20:59:41Z"
 content="""
@joey I don't understand you last comment where you state that special remotes can act as transfer repositories \"to transfer the files between computers that do not communicate directly\".  If there's no communication, ie git pushes or pulls, between the computers then they don't know what file names the files on the special remote map to. They need to somehow communicate the git repo too, don't they?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.63"
 subject="comment 11"
 date="2013-08-26T18:36:03Z"
 content="""
@Chris yes in that case you still need, a central git repository (which need not be on a host that supports git-annex), or the assistant can use xmpp to sync the git data.
"""]]


code:
[[!comment format=mdwn
 username="diepes"
 ip="105.227.34.36"
 subject="rsync description and example"
 date="2013-03-29T15:57:20Z"
 content="""
Hi, I would like to see a example of setting up / using e.g. rsync.net as a repo.

Please also provide a highlevel description of how the rsync repo fits in with git-annex.

Q1. can you adds files to the remote rsync repo, and will they be detected and synced back ?
Q2. is all the git history rsync'd to remote ?   how do i recover if i loose all data except the remote rsync repo ?

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 2"
 date="2013-03-29T17:12:47Z"
 content="""
No, special remotes do not contain a copy of the git repository, and no, git-annex does not notice when files are added to remote rsync repositories. Suggest you read [[special_remotes]].
"""]]


code:
[[!comment format=mdwn
 username="diepes"
 ip="105.227.34.36"
 subject="sshfs ?  and no sync from special remote to 2nd git-annex ?"
 date="2013-03-29T22:09:49Z"
 content="""
* It sound to me it would be 1st prize if the cloud provider supported the git-annex functionality over ssh.
* Then it could be a full git-annex repo, and used for recovery if my laptop with the git info gets lost.
* From special remotes \"These can be used just like any normal remote by git-annex\"
* Your comment \"No, special remotes do not contain a copy of the git repository\"

 * so a special remote is

1. \"A. just a remote filesystem, that contains the objects with sha1 names ? \"
2. \"B. there is no git info and thus no file name & directory details or am i missing something ?\"
3. \"C. you can't use the remote as a cloud drive to sync changes e.g. file moves, renames between two other git-annex repositories ? (no meta data)\"
4. \"D. the data on the cloud/rsync storage can not be used directly it has to moved into a git-annex capable storage location. \"

* Would it not be better to mount the remote storage over ssh(sshfs) and then use full git-annex on mounted directory ?
"""]]


code:
[[!comment format=mdwn
 username="diepes"
 ip="41.160.8.49"
 subject="rsync.net support git and rsync, not git-annex"
 date="2013-05-13T14:44:53Z"
 content="""
how would i use rsync.net as a full repository ?  (annex files and git repo)

It support's rsync, and I discovered now git as well.

I had a look at the webapp and can't figure out how to set it up, so I can have multiple remote's sync to rsync.net for annex files and git sync.

Link: http://www.rsync.net/resources/howto/github_integration.html

"""]]


code:
[[!comment format=mdwn
 username="cehteh"
 ip="217.8.62.137"
 subject="rsync daemon mode"
 date="2013-07-27T01:35:37Z"
 content="""
rsync has a --daemon mode with a simple challenge-response authentication but no encryption. This offers a nice lightweight alternative to ssh, especially when we
store/transfer encrypted content anyways. Is this already supported in git-annex, if yes how to set it up?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.246.110"
 subject="comment 6"
 date="2013-07-28T00:11:38Z"
 content="""
The first example on this page shows using rsync:// to store files on a system using the rsync daemon.
"""]]


code:
[[!comment format=mdwn
 username="http://olivier.berger.myopenid.com/"
 nickname="obergix"
 subject="rsync remote is basically more intended for backups ?"
 date="2013-08-17T17:40:47Z"
 content="""
If I get it correctly, it is mainly useable as a backup, which will accumulate contents of the objects managed by git-annex over time.

It would be great to have a use case illustrating its use in concrete matters. Thanks in advance.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.63"
 subject="comment 8"
 date="2013-08-22T18:02:00Z"
 content="""
There are many use cases for a rsync special remote. You could use it as a backup. You could use it to archive files offline in a drive with encryption enabled so if the drive is stolen your data is not. You could `git annex move --to rsyncremote` large files when your local drive is getting full, and then `git annex move` the files back when free space is again available. You could have one repository copy files to a rsync remote, and then `git annex get` them on another repository, to transfer the files between computers that do not communicate directly. The git-annex assistant makes it easy to set up rsync remotes using this last scenario, which is referred to as a transfer repository, and arranges to drop files from the transfer repository once they have been transferred to all known clients.

None of these use cases are particular to rsync remotes. Most special remotes can all be used in these and other ways. It largely doesn't matter for your use what underlying transport the special remote uses.
"""]]


code:
[[!comment format=mdwn
 username="http://olivier.berger.myopenid.com/"
 nickname="obergix"
 subject="Added use cases to "special remotes""
 date="2013-08-22T20:23:13Z"
 content="""
Thanks @joeyh. I've taken the liberty to add your use case description to [[special remotes]]. Hope this helps.
"""]]


code:
This special remote stores file contents using
[Tahoe-LAFS](http://tahoe-lafs.org/). There are a number of commercial
providers, or you can build your own tahoe storage grid.

Since Tahoe-LAFS encrypts all data stored in it, git-annex does not do any
additional encryption of its own.

Note that data stored in a tahoe remote cannot be dropped from it, as
Tahoe-LAFS does not support removing data once it is stored in the Tahoe grid.
This, along with Tahoe's ability to recover data when some nodes fail,
makes a tahoe special remote an excellent choice for storing backups.

Typically you will have an account on a Tahoe-LAFS storage grid, which
is represented by an "introducer furl". You need to supply this to
git-annex in the `TAHOE_FURL` environment variable when initializing the
remote. git-annex will then generate a tahoe configuration directory for
the remote under `~/.tahoe/git-annex/`, and automatically start the tahoe
daemon as needed.

## configuration

These parameters can be passed to `git annex initremote` to configure
the tahoe remote.

* `shared-convergence-secret` - Optional. Can be useful to set to
  allow tahoe to deduplicate information. By default, a new
  shared-convergence-secret is created for each tahoe remote.

* `embedcreds` - Optional. Set to "yes" embed the tahoe credentials
  (specifically the introducer-furl and shared-convergence-secret)
  inside the git repository, which allows other clones to also use them
  in order to access the tahoe grid.

  Think carefully about who can access your git repository, and
  whether you want to give them access to your tahoe system before
  using embedcreds!

Setup example:

	# TAHOE_FURL=... git annex initremote tahoe type=tahoe embedcreds=yes

## old version of tahoe special remote

An older implementation of tahoe for git-annex used
the hook special remote. It is not compatible with this newer
implementation. See
[[tahoe-lafs|forum/tips:_special__95__remotes__47__hook_with_tahoe-lafs]].


code:
git-annex can use the WWW as a special remote, downloading urls to files.
See [[tips/using_the_web_as_a_special_remote]] for usage examples.

## notes

Currently git-annex only supports downloading content from the web;
it cannot upload to it or remove content.

This special remote uses arbitrary urls on the web as the source for content.
git-annex can also download content from a normal git remote, accessible by
http.


code:
[[!comment format=mdwn
 username="http://olivier.berger.myopenid.com/"
 nickname="obergix"
 subject="Which URL prefix are supported ?"
 date="2013-08-17T08:44:05Z"
 content="""
It is not clear whether only http:// URLs are supported. Can you list others ?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4978:f:21a::2"
 subject="comment 2"
 date="2013-08-17T08:59:11Z"
 content="""
When it says \"arbitrary urls\", it means it. The only requirement is that the url be well formed and that wget or whatever command you have it configured to use via annex.web-download-command knows how to download it.
"""]]


code:
This special remote type stores file contents in a WebDAV server.

## configuration

The environment variables `WEBDAV_USERNAME` and `WEBDAV_PASSWORD` are used
to supply login credentials. You need to set these only when running
`git annex initremote`, as they will be cached in a file only you
can read inside the local git repository.

A number of parameters can be passed to `git annex initremote` to configure
the webdav remote.

* `encryption` - One of "none", "hybrid", "shared", or "pubkey".
  See [[encryption]].

* `keyid` - Specifies the gpg key to use for [[encryption]].

* `embedcreds` - Optional. Set to "yes" embed the login credentials inside
  the git repository, which allows other clones to also access them. This is
  the default when gpg encryption is enabled; the credentials are stored
  encrypted and only those with the repository's keys can access them.

  It is not the default when using shared encryption, or no encryption.
  Think carefully about who can access your repository before using
  embedcreds without gpg encryption.

* `url` - Required. The URL to the WebDAV directory where files will be
  stored. This can be a subdirectory of a larger WebDAV repository, and will
  be created as needed. Use of a https URL is strongly
  encouraged, since HTTP basic authentication is used.

* `chunk` - Enables [[chunking]] when storing large files.

* `chunksize` - Deprecated version of chunk parameter above.
  Do not use for new remotes. It is not safe to change the chunksize
  setting of an existing remote.

Setup example:

	# WEBDAV_USERNAME=joey@kitenet.net WEBDAV_PASSWORD=xxxxxxx git annex initremote box.com type=webdav url=https://dav.box.com/dav/git-annex chunksize=75mb keyid=joey@kitenet.net


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnq-RfkVpFN15SWvQ2lpSGAi0XpNQuLxKM"
 nickname="Yuval"
 subject="re: Change Password"
 date="2014-05-04T19:09:55Z"
 content="""
I have Carl's problem as well. Changed box.com's password and now things are crazy.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 11"
 date="2014-05-16T18:19:21Z"
 content="""
You can change the webdav password by running:

WEBDAV_USERNAME=whatever WEBDAV_PASSWORD=newpassword git annex enableremote $remotename
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnmF_9CAtfqdZkC4e-_dCX-rK5bqh4RWkw"
 nickname="Carl"
 subject="comment 12"
 date="2014-05-17T20:47:00Z"
 content="""
I tried it, but I do have som troubles. Changing password seemed to have worked. At least I do think it did say OK. Any further action generates the following response:

$ git annex fsck --from box.com3 Literatur/Science/Papers/Modén/2006.pdf

fsck Literatur/Science/Papers/Modén/2006.pdf (gpg) (checking box.com3...) (StatusCodeException (Status {statusCode = 401, statusMessage = \"Unauthorized\"}) [(\"Server\",\"nginx\"),(\"Date\",\"Sat, 17 May 2014 19:04:51 GMT\"),(\"Content-Type\",\"application/xml; charset=utf-8\"),(\"Content-Length\",\"235\"),(\"Connection\",\"keep-alive\"),(\"Vary\",\"Host\"),(\"WWW-Authenticate\",\"Basic realm=\\"dav.box.com\\"\"),(\"X-Response-Body-Start\",\"<?xml version=\\"1.0\\" encoding=\\"utf-8\\"?>\n<d:error xmlns:d=\\"DAV:\\" xmlns:s=\\"http://sabredav.org/ns\\">\n  <s:exception>Sabre_DAV_Exception_NotAuthenticated</s:exception>\n  <s:message>Username or password does not match</s:message>\n</d:error>\n\")] (CJ {expose = []})) failed
git-annex: fsck: 1 failed

It seems it is not available to use the correct password. I tried to use initremote and create a new remote, and this works. In the end no data is lost, as three copies out of my four are still there. I will keep my old remote around for some time if there is any sort of errorfinding that can be done.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmZilYULa6CDEGfuagoDlesyakBgnf-dF8"
 nickname="Maarten"
 subject="Box.com and unknown CA?"
 date="2014-05-21T04:07:36Z"
 content="""
When trying to enable box.com for jabber sharing, I get:

    WebDAV failed to write file: TlsException (HandshakeFailed (Error_Protocol (\"certificate has unknown CA\",True,UnknownCa))): user error

How can I debug this situation?
"""]]


code:
[[!comment format=mdwn
 username="http://rfhbuk.myopenid.com/"
 ip="109.145.123.81"
 subject="git-annex initremote failing for webdav servers"
 date="2012-12-01T10:18:04Z"
 content="""
Unfortunately, trying to set up the following webdav servers fail. Some of the terminal output of git-annex --debug initremote ... is below, but it may not really helpful. Can I help any further, can a webdav remote be set up in a different way? (The box.com webdav initremote worked fine.) Cheers

With livedrive.com:

    [2012-12-01 10:15:12 GMT] chat: gpg [\"--batch\",\"--no-tty\",\"--use-agent\",\"--quiet\",\"--trust-model\",\"always\",\"--encrypt\",\"--no-encrypt-to\",\"--no-default-recipient\",\"--recipient\",\"xxxxxxxxxxxxxx\"]
    (encryption setup with gpg key xxxxxxxxxxxxxx) (testing WebDAV server...)
    git-annex: \"Bad Request\": user error
    failed
    git-annex: initremote: 1 failed

With sd2dav.1und1.de:

    [2012-12-01 10:13:10 GMT] chat: gpg [\"--batch\",\"--no-tty\",\"--use-agent\",\"--quiet\",\"--trust-model\",\"always\",\"--encrypt\",\"--no-encrypt-to\",\"--no-default-recipient\",\"--recipient\",\"xxxxxxxxxxxxxx\"]
    (encryption setup with gpg key xxxxxxxxxxxxxx) (testing WebDAV server...)
    git-annex: \"Locked\": user error
    failed
    git-annex: initremote: 1 failed
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.6.49"
 subject="comment 2"
 date="2012-12-01T18:34:05Z"
 content="""
I tried signing up for livedrive, but I cannot log into it with WebDav at all. Do they require a Pro account to use WebDav?

When it's \"testing webdav server\", it tries to make a collection (a subdirectory), and uploads a file to it, and sets the file's properties, and deletes the file. One of these actions must be failing, perhaps because the webdav server implementation does not support it. Or perhaps because the webdav client library is doing something wrong. I've instrumented the test, so it'll say which one.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.6.49"
 subject="comment 3"
 date="2012-12-01T20:13:36Z"
 content="""
I've identified the problem keeping it working with livedrive. Once a patch I've written is applied to the Haskell DAV library, I'll be able to update git-annex to support it.

I don't know about sd2dav.1und1.de. The error looks like it doesn't support WebDAV file locking.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.6.49"
 subject="comment 4"
 date="2012-12-01T21:14:34Z"
 content="""
Good news, livedrive is supported now, by git-annex's master branch.
"""]]


code:
[[!comment format=mdwn
 username="http://peter-simons.myopenid.com/"
 ip="192.100.130.7"
 subject="WebDAV without locking"
 date="2013-01-16T12:44:16Z"
 content="""
Hi Joey,

you are right, the 1-und-1.de implementation of WebDAV does not support locking.

Do you think there is a way to make that service work with git-annex anyhow -- say, by allowing the user to disable file-locking? I've worked around the problem with 1-und-1.de by using fuse+davfs2 to mount the storage in my file system and access it through the directory backend, but FUSE is dead slow, unfortunately, and this solution really sucks.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.7.238"
 subject="comment 6"
 date="2013-01-17T18:23:27Z"
 content="""
It seems it must advertise it supports the LOCK and UNLOCK http actions, but fails when they're used.

The DAV library I am using always locks if it seems the server supports it. So this will need changes to that library. I've filed a bug requesting the changes. <http://bugs.debian.org/698379>
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawniayrgSdVLUc3c6bf93VbO-_HT4hzxmyo"
 nickname="Tobias"
 subject="SSL certificates"
 date="2013-03-25T16:09:54Z"
 content="""
Trying to use webdav leads in:

    git-annex: HandshakeFailed (Error_Protocol (\"certificate rejected: not coping with anything else Start (Container Context 0)\",True,CertificateUnknown))

How can I disable the SSL certificate check?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 8"
 date="2013-03-27T16:38:47Z"
 content="""
I don't think the checking of SSL certificates can currently be disabled.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnmF_9CAtfqdZkC4e-_dCX-rK5bqh4RWkw"
 nickname="Carl"
 subject="Change Password"
 date="2014-05-02T12:30:58Z"
 content="""
I want to change the password that I use on a webdav service (Heartbleed mitigration). What do I need to do to in git annex to make it have access to the new password? Thanks in anvance.
"""]]


code:
XMPP (Jabber) is used by the [[assistant]] as a git remote. This is,
technically not a git-annex special remote (large files are not transferred
over XMPP; only git commits are sent).

Typically XMPP will be set up using the web app, but here's how a manual
set up could be accomplished:

1. xmpp login credentials need to be stored in `.git/annex/creds/xmpp`.
   Obviously this file should be mode 600. An example file:

	XMPPCreds {xmppUsername = "joeyhess", xmppPassword = "xxxx", xmppHostname = "xmpp.l.google.com.", xmppPort = 5222, xmppJID = "joeyhess@gmail.com"}

2. A git remote is created using a special url, of the form `xmpp::user@host`
   For the above example, it would be `url = xmpp::joeyhess@gmail.com`

3. The uuid of one of the other clients using XMPP should be configured
   using the `annex.uuid` setting, the same as is set up for other remotes.

With the above configuration, the [[assistant]] will use xmpp remotes much as
any other git remote. Since XMPP requires a client that is continually running
to see incoming pushes, the XMPP remote cannot be used with git at the
command line.

## XMPP server support status
[[!table  data="""
Provider|Status|Type|Notes
[[Gmail|http://gmail.com]]|Working|?|Google Apps: [setup your SRV records](http://www.olark.com/gtalk/check_srv) or configure `.git/annex/creds/xmpp` manually
[[Coderollers|http://www.coderollers.com/xmpp-server/]]|Working|[[Openfire|http://www.igniterealtime.org/projects/openfire/]]
[[jabber.me|http://jabber.me/]]|Working|[[Tigase|http://www.tigase.org/]]
[[xmpp.ru.net|https://www.xmpp.ru.net]]|Working|[[jabberd2|http://jabberd2.org/]]
[[jabber.org|http://jabber.org]]|Working|[[Isode M-Link|http://www.isode.com/products/m-link.html]]
-|Working|[[Prosody|http://prosody.im/]]|No providers tested.
-|Working|[[Metronome|http://www.lightwitch.org/]]|No providers tested.
-|[[Failing|http://git-annex.branchable.com/forum/XMPP_authentication_failure/]]|ejabberd|[[Authentication bug|https://support.process-one.net/browse/EJAB-1632]]: Fixed in debian unstable (>= 2.1.10-5) and stable (>=2.1.10-4+deb7u1)
-|[[Failing|http://git-annex.branchable.com/forum/XMPP_authentication_failure/#comment-4ce5aeabd12ca3016290b3d8255f6ef1]]|jabberd14|No further information
"""]]
List of providers: [[http://xmpp.net/]]

See also: [[xmpp_protocol_design_notes|design/assistant/xmpp]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmZilYULa6CDEGfuagoDlesyakBgnf-dF8"
 nickname="Maarten"
 subject="Google Apps & SRV"
 date="2013-11-27T18:13:18Z"
 content="""
To make XMPP via a Google Apps account work out of the box, all you need to do is add the correct SRV records to your domain's DNS zone.  If you do, you don't need to manually create creds/xmpp, you can just use the web interface, enter your full apps email address and password (remember, if 2-factor, this needs to be an application-specific password).

To create the correct SRV records (and check whether they're correct), see http://www.olark.com/gtalk/check_srv
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm0yDI6AsAuXPDHIDXm5H9EGDXkuwb1efA"
 nickname="Yann"
 subject="XMPP not working in Raspberry PI build"
 date="2013-12-27T23:16:23Z"
 content="""
Hi,

I have the same problem that users Carlo and RaspberryPie have: XMPP not working and no XMPP log in the build.

I did a quick strace during the daemon start and I noticed than the raspberry pi build doesn't even try to read the creds/xmpp files contrary to the standard git-annex package on my ubuntu workstation.

To me, it seems the XMPP thread is not started at all.
Does someone know any reason why the XMPP thread could not be started ?

If there is anything I can do to help debug this issue, let me know.

Yann

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmWg4VvDTer9f49Y3z-R0AH16P4d1ygotA"
 nickname="Tobias"
 subject="User defined server"
 date="2013-04-17T09:45:59Z"
 content="""
It would be nice if you could expand the XMPP setup in the assistant to support an \"advanced\" settings view where a custom server could be defined.

Example: I have a google Apps domain called mytest.com, with the users bla1@mytest.com and bla2@mytest.com. When trying to add either of those accounts to the assistant XMPP will try to use mytest.com as the jabber server, and not googles server.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmRFKwny4rArBaz-36xTcsJYqKIgdDaw5Q"
 nickname="Andrew"
 subject="comment 2"
 date="2013-04-17T22:28:39Z"
 content="""
Yeah, I agree, this would be nice.

For your own domain, you can configure DNS like this: [[http://support.google.com/a/bin/answer.py?hl=en&answer=34143]] to make XMPP find the right server. But for some that's not an option and the \"advanced\" mode would be useful in that case.
"""]]


code:
[[!comment format=mdwn
 username="RaspberryPie"
 ip="46.19.143.203"
 subject="Missing prerequisites for XMPP syncing?"
 date="2013-09-17T06:53:59Z"
 content="""
I set up two fresh annexes that can talk via XMPP and no other way. After I fire up the assistants I expect them to sync their metadata, but nothing happens. One log gives me an 'XMPPClient: received: [\"Unknown message\"]' message every two minutes. The other one doesn't contain the string XMPP at all, not once. So my suspicion is that this particular version of git-annex doesn't support XMPP, which is weird because:

    $ git annex version
    git-annex version: 4.20130909
    build flags: Assistant Pairing Testsuite S3 Inotify XMPP DNS Feeds
    local repository version: 3
    default repository version: 3
    supported repository versions: 3 4
    upgrade supported from repository versions: 0 1 2

This is the version output from the other machine:

    $ git annex version
    git-annex version: 4.20130827
    build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP
    local repository version: 3
    default repository version: 3
    supported repository versions: 3 4
    upgrade supported from repository versions: 0 1 2

What am I missing? Are there more build flags for XMPP than the one called XMPP? (Also, no, I can't just copy versions between machines b/c the architectures are different. And yep, the one giving me trouble is ARM.)
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.14.105"
 subject="comment 4"
 date="2013-09-19T21:07:35Z"
 content="""
If you have the XMPP flag in your git-annex build, it will support XMPP. Are you sure you set up the xmpp creds file and the xmpp special remote correctly on the ARM machine? (I assume it has no webapp, so you had to set that up manually..)

Here's how you can do that manually:

1. Run git-annex on a machine with the webapp, set up XMPP, and copy the .git/annex/creds/xmpp to the machine without the webapp.
2. On the machine without the webapp, add a git remote that has its \"url = xmpp::loginname@xmppserver.com\" and its annex-uuid set to the annex.uuid of the repository on the first machine.
3. Run git-annex assistant on the machine without the webapp.
"""]]


code:
[[!comment format=mdwn
 username="RaspberryPie"
 ip="37.221.161.234"
 subject="Nope"
 date="2013-09-24T22:05:55Z"
 content="""
Your guess is right, Joey, I'm configuring by hand as the ARM machine has no webapp. And yes, I'm mostly sure I set up everything correctly. The XMPP account is working, and my configuration of git-annex is all but identical to your example.

Here's what I do. First on the machine with the webapp:

    mkdir ~/test
    cd ~/test
    git init
    git annex init
    git annex webapp

I set up XMPP from within the webapp. The file ~/test/.git/annex/creds/xmpp is created with the correct credentials. (BTW: The file's default permissions are 620 instead of 600 - is that a bug?)

I add a file or two to the annex for good measure. Then, on the ARM machine:

    mkdir ~/test
    cd ~/test
    git init
    git annex init
    mkdir .git/annex/creds
    scp -2 webappmachine:~/test/.git/annex/creds/xmpp .git/annex/creds
    chmod 600 .git/annex/creds/xmpp
    git remote add webappmachine xmpp::login@server

The final step is to edit .git/config on the ARM machine. The [remote] section now looks like this:

    [remote \"webappmachine\"]
        url = xmpp::login@server
        fetch = +refs/heads/*:refs/remotes/webappmachine/*
        annex-uuid = aaaaaaaa-bbbb-cccc-dddddddddddd

where aaaaaaaa-bbbb-cccc-dddddddddddd is the return value of `git config --get annex.uuid` on the webapp machine.

I then run `git annex assistant` on the ARM machine and expect the two machines to synchronize their metadata, e.g. the number of knownn annex keys in the repo. But it doesn't happen.

So I set `debug = true`, restart the assistants and check the log. This is what I get on the webapp machine:

    [2013-09-24 17:45:41 EDT] XMPPClient: connected a5/25577ac4-3248-4c83-8391-bd93708bcf2b
    [2013-09-24 17:45:41 EDT] XMPPClient: received: [\"Presence from a5/dc9bcde8-fe18-47de-807c-c620019279f2 Just (Element {elementName = Name {nameLocalName = \\"git-annex\\", nameNamespace = Just \\"git-annex\\", namePrefix = Nothing}, elementAttributes = [(Name {nameLocalName = \\"query\\", nameNamespace = Nothing, namePrefix = Nothing},[ContentText \\"\\"])], elementNodes = []})\",\"QueryPresence\"]
    [2013-09-24 17:45:42 EDT] XMPPClient: received: [\"Presence from a5/900e3b6e-a7f4-4a6a-8d12-ed94de429258 Just (Element {elementName = Name {nameLocalName = \\"git-annex\\", nameNamespace = Just \\"git-annex\\", namePrefix = Nothing}, elementAttributes = [(Name {nameLocalName = \\"push\\", nameNamespace = Nothing, namePrefix = Nothing},[ContentText \\"43357474-abbb-4667-a334-e4615ea6d4a2\\"])], elementNodes = []})\",\"NotifyPush [UUID \\"43357474-abbb-4667-a334-e4615ea6d4a2\\"]\"]
    [2013-09-24 17:45:42 EDT] XMPPClient: push notification for
    [2013-09-24 17:45:42 EDT] read: git [\"--git-dir=/home/pi/test/.git\",\"--work-tree=/home/pi/test\",\"symbolic-ref\",\"HEAD\"]
    [2013-09-24 17:45:42 EDT] read: git [\"--git-dir=/home/pi/test/.git\",\"--work-tree=/home/pi/test\",\"show-ref\",\"refs/heads/master\"]
    [2013-09-24 17:45:42 EDT] XMPPClient: received: [\"Pushing \\"a59\\" (CanPush (UUID \\"d50c4cc9-e7c0-4ef0-84c6-f11012051eb9\\") [34f875cc7fa1198414f93990af9ab78e6cee893e,6fad42234060361435d6cf2ab4bd40e438c2d05c])\"]
    [2013-09-24 17:45:42 EDT] read: git [\"--git-dir=/home/pi/test/.git\",\"--work-tree=/home/pi/test\",\"show-ref\",\"git-annex\"]
    [2013-09-24 17:45:42 EDT] read: git [\"--git-dir=/home/pi/test/.git\",\"--work-tree=/home/pi/test\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
    [2013-09-24 17:45:42 EDT] read: git [\"--git-dir=/home/pi/test/.git\",\"--work-tree=/home/pi/test\",\"log\",\"refs/heads/git-annex..6fad42234060361435d6cf2ab4bd40e438c2d05c\",\"--oneline\",\"-n1\"]
    [2013-09-24 17:45:42 EDT] chat: git [\"--git-dir=/home/pi/test/.git\",\"--work-tree=/home/pi/test\",\"cat-file\",\"--batch\"]
    [2013-09-24 17:45:42 EDT] XMPPClient: received: [\"Ignorable Presence from a5/25577ac4-3248-4c83-8391-bd93708bcf2b Just (Element {elementName = Name {nameLocalName = \\"git-annex\\", nameNamespace = Just \\"git-annex\\", namePrefix = Nothing}, elementAttributes = [], elementNodes = []})\"]
    [2013-09-24 17:45:42 EDT] XMPPClient: received: [\"Unknown message\"]
    [2013-09-24 17:45:42 EDT] XMPPClient: received: [\"Pushing \\"a59\\" (PushRequest (UUID \\"d50c4cc9-e7c0-4ef0-84c6-f11012051eb9\\"))\"]
    [2013-09-24 17:45:42 EDT] XMPPSendPack: started running push Pushing \"a59\" (PushRequest (UUID \"d50c4cc9-e7c0-4ef0-84c6-f11012051eb9\"))
    [2013-09-24 17:45:42 EDT] read: git [\"--git-dir=/home/pi/test/.git\",\"--work-tree=/home/pi/test\",\"symbolic-ref\",\"HEAD\"]
    [2013-09-24 17:45:42 EDT] XMPPClient: received: [\"Ignorable Presence from a5/25577ac4-3248-4c83-8391-bd93708bcf2b Just (Element {elementName = Name {nameLocalName = \\"git-annex\\", nameNamespace = Just \\"git-annex\\", namePrefix = Nothing}, elementAttributes = [], elementNodes = []})\"]
    [2013-09-24 17:45:42 EDT] read: git [\"--git-dir=/home/pi/test/.git\",\"--work-tree=/home/pi/test\",\"show-ref\",\"refs/heads/master\"]
    [2013-09-24 17:45:42 EDT] call: git [\"--git-dir=/home/pi/test/.git\",\"--work-tree=/home/pi/test\",\"branch\",\"-f\",\"synced/master\"]
    [2013-09-24 17:45:42 EDT] XMPPSendPack: finished running push Pushing \"a59\" (PushRequest (UUID \"d50c4cc9-e7c0-4ef0-84c6-f11012051eb9\")) False

And from then on, in two-minute intervals:

    [2013-09-24 17:47:42 EDT] XMPPClient: received: [\"Unknown message\"]
    [2013-09-24 17:49:42 EDT] XMPPClient: received: [\"Unknown message\"]
    [2013-09-24 17:51:42 EDT] XMPPClient: received: [\"Unknown message\"]

The log on the ARM machine is rather unhelpful. Actually it doesn't even contain the string \"XMPP\". This looks to me like the webapp machine tries to communicate via Jabber but doesn't get any intelligible answer. And this is the reason I wondered whether the problem lies with my self-compiled ARM git-annex  binary. I actually spent a while compiling 4.20130909 with all flags but webapp and webdav, but the result is still the same.

Any other ideas what I'm doing wrong here?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.220"
 subject="comment 6"
 date="2013-09-25T18:13:24Z"
 content="""
If you're not getting an \"XMPPClient: connected\", then my guess would be that your git-annex build's XMPP is screwed up somehow. For example, if it hung forever when connecting to the XMPP server, it would never get as far as printing that message. (If it tried and failed to connect, you'd get a message about the connection having failed.)
"""]]


code:
[[!comment format=mdwn
 username="RaspberryPie"
 ip="46.165.221.166"
 subject="comment 7"
 date="2013-09-26T03:46:18Z"
 content="""
I see. Is there a way to check whether the build is corrupt? The build logs gave me nothing.

Anyway, XMPP is not the most important feature to me. It still bugs me though that it doesn't work when it should.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnZEanlyzay_QlEAL0CWpyZcRTyN7vay8U"
 nickname="Carlo"
 subject="comment 8"
 date="2013-10-22T14:30:58Z"
 content="""
Same setup, same problem... no log output on raspberry pi.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlu-fdXIt_RF9ggvg4zP0yBbtjWQwHAMS4"
 nickname="Jörn"
 subject="Same problem, no XMPP showing up in daemon.log"
 date="2013-11-21T21:13:16Z"
 content="""
I have the same setup like @RaspberryPie, except that my server is not running on the Pi but on Debian7-amd64. On my client (OSX, self-compiled using cabal) I can see XMPP log entries like @RaspberryPi, however, on the Debian7 machine (also self-compiled) I do not see any XMPP entry in the daemon.log. Setup regarding .git/annex/creds/xmpp and the special xmpp remote is correct (checked a thousand times).

Do you have any idea what could be wrong, Joey? Thanks a lot.

Output of git annex version:

    git-annex version: 5.20131120
    build flags: Assistant Pairing Testsuite S3 WebDAV Inotify DBus XMPP DNS Feeds Quvi CryptoHash
    key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
    remote types: git gcrypt S3 bup directory rsync web webdav glacier hook
    local repository version: 4
    default repository version: 3
    supported repository versions: 3 5
    upgrade supported from repository versions: 0 1 2 4

Jörn
"""]]


code:
git-annex allows managing files with git, without checking the file
contents into git. While that may seem paradoxical, it is useful when
dealing with files larger than git can currently easily handle, whether due
to limitations in memory, time, or disk space.

[[!img assistant/thumbnail.png link=assistant align=right]]
git-annex is designed for git users who love the command line.
For everyone else, the [[git-annex assistant|assistant]] turns
git-annex into an easy to use folder synchroniser.

To get a feel for git-annex, see the [[walkthrough]].


code:
The `git annex sync` command provides an easy way to keep several
repositories in sync.

Often git is used in a centralized fashion with a central bare repository
which changes are pulled and pushed to using normal git commands.
That works fine, if you don't mind having a central repository.

But it can be harder to use git in a fully decentralized fashion, with no
central repository and still keep repositories in sync with one another.
You have to remember to pull from each remote, and merge the appropriate
branch after pulling. It's difficult to *push* to a remote, since git does
not allow pushes into the currently checked out branch.

`git annex sync` makes it easier using a scheme devised by Joachim
Breitner. The idea is to have a branch `synced/master` (actually,
`synced/$currentbranch`), that is never directly checked out, and serves
as a drop-point for other repositories to use to push changes.

When you run `git annex sync`, it merges the `synced/master` branch
into `master`, receiving anything that's been pushed to it. (If there is a
conflict in this merge, [[automatic_conflict_resolution]] is used to
resolve it). Then it fetches from each remote, and merges in any changes that
have been made to the remotes too. Finally, it updates `synced/master`
to reflect the new state of `master`, and pushes it out to each of the remotes.

This way, changes propagate around between repositories as `git annex sync`
is run on each of them. Every repository does not need to be able to talk
to every other repository; as long as the graph of repositories is
connected, and `git annex sync` is run from time to time on each, a given
change, made anywhere, will eventually reach every other repository.

The workflow for using `git annex sync` is simple:

* Make some changes to files in the repository, using `git-annex`,
  or anything else.
* Run `git annex sync` to save the changes.
* Next time you're working on a different clone of that repository,
  run `git annex sync` to update it.

Note that by default, `git annex sync` only synchronises the git
repositories, but does not transfer the content of annexed files. If you
want to fully synchronise two repositories content,
you can use `git annex sync --content`. You can also configure
[[preferred_content]] settings to make only some content be synced.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkI9AR8BqG4RPw_Ov2lnDCJWMuM6WMRobQ"
 nickname="Dav"
 subject="Sorry to just be getting back..."
 date="2014-01-26T22:51:28Z"
 content="""
The URLs in question in this case were read-only github https URLs. In any case, my problems are solved by what you've already suggested. I think a less error-sounding response to read-only https repos sounds nice!
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn3p4i4lk_zMilvjnJ9sS6g2nerpgz0Fjc"
 nickname="Matthias"
 subject="sync slow with content switch"
 date="2014-04-22T20:37:05Z"
 content="""
I noticed that in a test with 2 local repositories and around 2'000 files \"git annex sync\" is still very fast, but \"git annex sync --content\" takes multiple seconds. Is this avoidable?

I have a central repo and client repos. I want to copy all content to the central repo after a commit. Right now, I use \"git annex group central backup\", \"git annex wanted central standard\", and a hook that triggers \"git annex sync --content\" after each commit. Maybe there is a more efficient way to do this? Thanks for sharing thoughts.
"""]]


code:
[[!comment format=mdwn
 username="http://mshri.livejournal.com/"
 ip="129.170.212.202"
 subject="Sync specific branch or ignore a branch during sync"
 date="2014-04-25T15:37:53Z"
 content="""

I too feel that syncing all remotes by default is the right thing to do, but I think it should be limited to the 'master' and 'git-annex' branch. I often create branches that I want to keep local and do not want them to be synced. But I want 'master' and 'git-annex' branches to be synced with all remotes.

So it would be nice to able to set an option to sync all branches or just the 'master' and 'git-annex' or to able to ignore some branches during git annex sync

Shri

"""]]


code:
[[!comment format=mdwn
 username="zardoz"
 ip="134.147.14.84"
 subject="comment 13"
 date="2014-05-15T08:28:09Z"
 content="""
I agree with mshri. It’s confusing to have every local branch wind up on every remote (and it hinders «git annex unused»).

I tried working around this by just including relevant branches in the «fetch» refspec, but this will only work until another remote pushes the branches again.
"""]]


code:
[[!comment format=mdwn
 username="zardoz"
 ip="134.147.14.84"
 subject="comment 14"
 date="2014-05-15T08:58:26Z"
 content="""
Added a wishlist item <http://git-annex.branchable.com/todo/Allow_syncing_only_selected_branches/>
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 15"
 date="2014-05-15T19:53:16Z"
 content="""
We seem to have some rumor going around that `git annex sync` pushes all branches. It does not. It pushes only the git-annex branch and the currently checked out branch.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 16"
 date="2014-05-15T19:54:54Z"
 content="""
@Matthias, `git annex sync --content` has to check each file to see if any other repository wants it. This is necessarily going to get slow when there are a lot of files. The assistant does a similar syncing but uses some tricks to avoid scanning all the files too often, while still managing to keep them all in sync -- it can do this since it's a long-running daemon and is aware when files have changed.
"""]]


code:
[[!comment format=mdwn
 username="http://adamspiers.myopenid.com/"
 nickname="Adam"
 subject="very nice"
 date="2012-02-25T15:02:18Z"
 content="""
Here's a way to get from a starting point of two or more peer directory trees *not* tracked by git or git-annex, to the point where they can be synced in the manner described above: [[forum/syncing_non-git_trees_with_git-annex/]]
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlDDW-g2WLLsLpcnCm4LykAquFY_nwbIrU"
 nickname="Daniel"
 subject="Its just grand"
 date="2013-01-04T14:45:35Z"
 content="""
I cam upon git-annex a few months ago. I saw immidiately how it could help with some frustrations I've been having. One in particlar is keeping my vimrc in sync accross multiple locations and platforms. I finally took the time to give it a try after I finally hit my boiling point this morning. I went through the [walkthrough](http://git-annex.branchable.com/walkthrough/) and now I have an annax everywhere I need it. `git annex sync` and my vimrc is up-to-date, simply grand!

Thanks so much for making git-annex,
[Daniel Wozniak](http://woz.io)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnVX_teMpJeQkyheivad_1XSCFSjpiWgx8"
 nickname="Diggory"
 subject="synchronising stored files with a bare repository"
 date="2013-01-11T16:52:38Z"
 content="""
Good for syncing indexes, but if I want to synchronise all data files too (specifically pushing to a remote bare repository), how do I do that?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.7.238"
 subject="comment 4"
 date="2013-01-11T18:18:07Z"
 content="""
Yes, sync only syncs the git branches, not git-annex data. To sync the date, you can run a command such as `git annex copy --to bareremote`. You could run that in cron. Or, the [[assistant]] can be run as a daemon, and automatically syncs git-annex data.
"""]]


code:
[[!comment format=mdwn
 username="http://elmimmo.myopenid.com/"
 nickname="chocolate.camera"
 subject="How to sync content with git-annex, not assistant"
 date="2013-10-11T09:58:12Z"
 content="""
Sure assistant can sync git-annex data across remotes. But how do I tell a repo to sync git-annex data, but not so manually as to having to know what exactly needs to be copied from/to where?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkI9AR8BqG4RPw_Ov2lnDCJWMuM6WMRobQ"
 nickname="Dav"
 subject="Syncing only a specific branch"
 date="2013-11-24T17:48:22Z"
 content="""
By default, `git annex sync` will sync to all remotes, unless you specify a remote. So, I have to specify, e.g., `git annex sync origin`. I can simplify this with aliases, I suppose, but I do a lot of teaching non-programmer scientists... so it'd be nice to be able to configure this (so beginning users don't have to keep track of as many things).

Is there (or will there be) a way to do this?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.64"
 subject="comment 7"
 date="2013-11-26T20:08:33Z"
 content="""
I feel that syncing with all remotes by default is the right thing for git annex sync to do.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkI9AR8BqG4RPw_Ov2lnDCJWMuM6WMRobQ"
 nickname="Dav"
 subject="Use case for not syncing to all remotes"
 date="2013-12-08T19:20:26Z"
 content="""
Just in case you haven't considered such a scenario - maybe you have suggestions for how to collaborate more effectively with git annex (and avoid warning messages):

I'm trying to teach beginning scientist programmers (mostly graduate students), and a common scenario is to fork some scientific code. I'd like forking on github to be mundane, and not trigger warnings, and generally have as little for folks to explicitly keep track of as possible (this seems to be a common concern we share, which leads you to prefer syncing to all remotes without the option to configure the default behavior!).

However, I am currently working with students on forking and fixing up scientific code where the upstream maintainer doesn't want to allow pushes upstream, except via pull request. So, part of our approach is to set up some common shared datasets in git annex (and these just end up in our fork). If we have an \"upstream\" remote, git annex will try to sync with it, and report an error.

So - that's why I'd like to be able to configure the deactivation of syncing to a defined branch (e.g., \"upstream\"). However, if you have other suggestions to smooth the workflow, I would also like to hear those!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="comment 9"
 date="2013-12-12T17:54:55Z"
 content="""
@Dav what kind of url does the upstream remote have? Perhaps it would be sufficient to make sync skip trying to push to git:// and http[s]:// remotes. Both are unlikely to accept pushes and in the cases where they do accept pushes it would be fine to need a manual `git push`.

Anyway, you can already configure which remotes get synced with. From the man page:

<pre>
       remote.<name>.annex-sync
              If set to false, prevents  git-annex  sync  (and  the  git-annex
              assistant) from syncing with this remote.
</pre>

So `git config remote.upstream.annex-sync=false`
"""]]


code:
<TMPL_IF PERMALINK>
<a href="<TMPL_VAR PERMALINK>"><TMPL_VAR TITLE>
<TMPL_ELSE>
<a href="<TMPL_VAR PAGEURL>"><TMPL_VAR TITLE>
</TMPL_IF>
<TMPL_IF TAGS>
<TMPL_LOOP TAGS>
 [<TMPL_VAR LINK>]
</TMPL_LOOP>
</TMPL_IF>

Posted <TMPL_VAR CTIME>
<TMPL_IF AUTHOR>
by
<TMPL_IF AUTHORURL>
<a href="<TMPL_VAR AUTHORURL>"><TMPL_VAR AUTHOR>
<TMPL_ELSE>
<TMPL_VAR AUTHOR>
</TMPL_IF>

</TMPL_IF>


code:
### Please describe the problem.

### What steps will reproduce the problem?

### What version of git-annex are you using? On what operating system?

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

# End of transcript or log.
"""]]


code:
<a href="<TMPL_VAR PAGEURL>"><TMPL_VAR TITLE>

<TMPL_VAR CONTENT>


code:
Git annex. Brilliant. Powerful. git-annex.branchable.com
— Gert van Dijk (@gertvdijk) April 1, 2012

git-annex is like dropbox without the chrome. git-annex.branchable.com
— Wil Chung (@iamwil) March 31, 2012

blender 2.62 + ffmpeg + dnxhd + git-annex = who needs final cut?
— cdotwright (@cdotwright) April 4, 2012

I want #git-annex whereis for all the stuff (not) in my room.
— topr (@derwelle) February 22, 2012

Git-annex now auto-syncing photos from my android phone to a Tor hidden SSH service I control (via @guardianproject's Orbot) #prismbreak
— Richard King (@graphiclunarkid) June 20, 2013

What excites me about GIT ANNEX is how it fundamentally tracks the
backup and availability of any data you own, and allows you to share
data with a large or small audience, ensuring that the data survives.

-- Jason Scott <http://ascii.textfiles.com/archives/3625>

Seen on IRC:

oh my god, git-annex is amazing
this is the revolution in fucking with gigantic piles of files that I've been waiting for

And then my own story: I have a ton of drives. I have a lot of servers. I
live in a cabin on **dialup** and often have 1 hour on broadband in a week
to get everything I need. Without git-annex, managing all this would not be
possible. It works perfectly for me, not a surprise since I wrote it, but
still, it's a different level of "perfect" than anything I could put
together before. --[[Joey]]

See also: [[design/assistant/blog/day_288__success_stories]]


code:
[[!comment format=mdwn
 username="gjost"
 ip="108.205.152.179"
 subject="distributed digital repository"
 date="2013-11-14T00:39:49Z"
 content="""
I am in the process of building a distributed digital repository using Git and git-annex as a foundation. Our project is to collect and preserve archival-quality audio and video from multiple bandwidth-challenged remote sites. We're using git-annex to help us sneakernet the metadata and binaries around and to maintain multiple copies in geographically separate locations. Git very neatly addresses the problem of tracking changes, while git-annex makes it possible to track the flow of information through a network of distributed repositories.
"""]]