code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm3uJkdiJJejvqix9dULvw_Ma7DCtB-6zA"
 nickname="Ian"
 subject="git clone git://git-annex.branchable.com/ gives an error"
 date="2012-08-13T20:57:34Z"
 content="""
Thought you would want to know

error: unable to create file doc/forum/__91__Installation__93___base-3.0.3.2_requires_syb___61____61__0.1.0.2_however_syb-0.1.0.2_was_excluded_because_json-0.5_requires_syb___62____61__0.3.3.mdwn (File name too long)
fatal: cannot create directory at 'doc/forum/__91__Installation__93___base-3.0.3.2_requires_syb___61____61__0.1.0.2_however_syb-0.1.0.2_was_excluded_because_json-0.5_requires_syb___62____61__0.3.3': File name too long

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 subject="comment 2"
 date="2012-08-16T23:28:30Z"
 content="""
Ok, I've renamed that long-ish filename.
"""]]

repo_name: kakafarm/git

license: unknown


code:
You can `cabal configure -fEKG` to build a git-annex that includes
the EKG remote monitoring interface.

To access the EKG control panel, go to
<http://localhost:4242/> while a git-annex command is running.

This EKG build is mostly useful for debugging resource usage problems.

[[!img ekg.png caption="git-annex webapp startup, and assistant startup scan"]]

Note that since only one process can open port 4242 at a time, running
more than one git-annex process with EKG support at the same time can
result in some "resource busy (Address already in use)" messages -- but
git-annex will continue to work.

----

## full profiling

For the really tricky memory leaks, here's how to make a profiling build of
git-annex.

1. `cabal configure` with only the flags you really need
2. `cabal build --ghc-options="-prof -auto-all -caf-all"`
  This will probably fail due to some missing profiling libraries.
  You have to get the profiling versions of all needed haskell libraries
  installed somehow.
3. Run git-annex with the special flags `+RTS -hc -p`
4. Reproduce the memory leak problem.
5. If the assistant was run, stop it.
6. `hp2ps -e8in -c git-annex.hp` to generate a .ps graph of memory usage.


code:
[[!toc]]

git-annex mostly does not use encryption. Anyone with access to a git
repository can see all the filenames in it, its history, and can access
any annexed file contents.

Encryption is needed when using [[special_remotes]] like Amazon S3, where
file content is sent to an untrusted party who does not have access to the
git repository.

Such an encrypted remote uses strong ([[symmetric|design/encryption]] or
asymmetric) encryption on the contents of files, as well as HMAC hashing
of the filenames. The size of the encrypted files, and access patterns
of the data, should be the only clues to what is stored in such a
remote.

You should decide whether to use encryption with a special remote before
any data is stored in it. So, `git annex initremote` requires you
to specify "encryption=none" when first setting up a remote in order
to disable encryption. To use encryption, you run
`git-annex initremote` in one of these ways:

* `git annex initremote newremote type=... encryption=hybrid keyid=KEYID ...`
* `git annex initremote newremote type=... encryption=shared`
* `git annex initremote newremote type=... encryption=pubkey keyid=KEYID ...`

## hybrid encryption keys

The [[hybrid_key_design|design/encryption]] allows additional
encryption keys to be added on to a special remote later. Due to this
flexibility, it is the default and recommended encryption scheme.

	git annex initremote newremote type=... [encryption=hybrid] keyid=KEYID ...

Here the KEYID(s) are passed to `gpg` to find encryption keys.
Typically, you will say "keyid=2512E3C7" to use a specific gpg key.
Or, you might say "keyid=joey@kitenet.net" to search for matching keys.

To add a new key and allow it to access all the content that is stored
in the encrypted special remote, just run `git annex
enableremote` specifying the new encryption key:

	git annex enableremote myremote keyid+=788A3F4C

While a key can later be removed from the list, note that
it will **not** necessarily prevent the owner of the key
from accessing data on the remote (which is by design impossible to prevent,
short of deleting the remote). In fact the only sound use of `keyid-=` is
probably to replace a revoked key:

	git annex enableremote myremote keyid-=2512E3C7 keyid+=788A3F4C

See also [[encryption_design|design/encryption]] for other security
risks associated with encryption.

## shared encryption key

Alternatively, you can configure git-annex to use a shared cipher to
encrypt data stored in a remote. This shared cipher is stored,
**unencrypted** in the git repository. So it's shared among every
clone of the git repository.

	git annex initremote newremote type=... encryption=shared

The advantage is you don't need to set up gpg keys. The disadvantage is
that this is **insecure** unless you trust every clone of the git
repository with access to the encrypted data stored in the special remote.

## regular public key encryption

This alternative simply encrypts the files in the special remotes to one or
more public keys. It might be considered more secure due to its simplicity
and since it's exactly the way everyone else uses gpg.

	git annex initremote newremote type=.... encryption=pubkey keyid=KEYID ...

A disavantage is that it is not easy to later add additional public keys
to the special remote. While the `enableremote` parameters `keyid+=` and
`keyid-=` can be used, they have **no effect** on files that are already
present on the remote. Probably the only use for these parameters is
to replace a revoked key:

	git annex enableremote myremote keyid-=2512E3C7 keyid+=788A3F4C

But even in this case, since the files are not re-encrypted, the revoked
key has to be kept around to be able to decrypt those files.
(Of course, if the reason for revocation is
that the key has been compromised, it is **insecure** to leave files
encrypted using that old key, and the user should re-encrypt everything.)

(Because filenames are MAC'ed, a cipher still needs to be
generated (and encrypted to the given key IDs).)

## MAC algorithm

The default MAC algorithm to be applied on the filenames is HMACSHA1. A
stronger one, for instance HMACSHA512, can be chosen upon creation
of the special remote with the option `mac=HMACSHA512`. The available
MAC algorithms are HMACSHA1, HMACSHA224, HMACSHA256, HMACSHA384, and
HMACSHA512. Note that it is not possible to change algorithm for a
non-empty remote.


code:
[[!comment format=mdwn
 username="zooko"
 ip="75.220.153.232"
 subject="Tahoe-LAFS comes with encryption"
 date="2011-05-18T04:32:14Z"
 content="""
The Tahoe-LAFS special remote automatically encrypts and adds cryptography integrity checks/digital signatures. For that special remote you should not use the git-annex encryption scheme.

Tahoe-LAFS encryption generates a new independent key for each file. This means that you can share access to one of the files without thereby sharing access to all of them, and it means that individual files can be deduplicated among multiple users.
"""]]


code:
Aggregating git-annex mentions from elsewhere on the net..

* [[!aggregate expirecount=25 name="twitter" feedurl="http://tmp.kitenet.net/git-annex-twitter.rss" url="http://search.twitter.com/search.atom?q=git-annex"]]


code:
### Recent [[news]]

[[!inline pages="news/* and !*/Discussion" archive=yes show=2 feeds=no]]

### [[devblog|devblog]]

[[!inline pages="devblog/* and !*/Discussion" archive=yes show=5 feeds=no]]


code:
### Recent [[videos]]

[[!inline pages="videos/* and !*/Discussion" archive=yes show=2 feeds=no]]

### Recent [[forum posts|forum]]

[[!inline pages="forum/* and !*/Discussion" archive=yes show=5 feeds=no]]


code:
This is a place to discuss using git-annex.
If you need help, advice, or anything, post about it here.

But, please don't post bug reports here. Put them in [[bugs]].
And please don't make wishlist requests here. Put them in [[todo]].
(If you post bugs/todo here, it'll just get moved.)

[[!inline pages="forum/* and !*/Discussion" archive=yes rootpage=forum postformtext="Add a new thread titled:"]]


code:
I have 2 clients (laptops) that  would (usually) not be online at the same time. Is it possible to use git-annex for syncing these 2 clients through an encrypted server?

I have tried to set this up with the assistant by using an USB-disk for copying the repository from one client to the other. The server is configured through the assistant as repository group 'transfer'. Now both clients see the encrypted ssh-server, each client copy stuff to the encrypted server, but no files are copied between the 2 clients. (In my case, running git-annex on the server is not an option. I have read through this forum, but it doesn't seem anyone has been able to get it to work.)


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnvVfFLW4CTKs7UjdiLIsOn_cxj1Jnh64I"
 nickname="Charl"
 subject="Change the encrypted server repository group to "full backup""
 date="2014-03-23T21:00:01Z"
 content="""
Have you tried changing the repository group of the encrypted server to \"full backup\" instead of \"transfer\"?

I've just started experimenting with git-annex, and this setup (two possibly remote laptops, one encrypted server) seems to be working after very limited testing. I'm currently importing a larger collection of about 40000 files, will see if it still does (it seems whilst one client is transferring to the remote server, the other client laptop is not beeing synced; it did sync after my first batch of copying however.)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkvN0eFnwvgw2JyTSSHw0QouytcxtPLln8"
 nickname="Stein Roald"
 subject="No success"
 date="2014-03-26T19:43:24Z"
 content="""
Thank you, Charl, for your suggestion. I've tried it, without success. (It doesn't seem like that change changed anything on the encrypted ssh-server.)
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.41"
 subject="comment 3"
 date="2014-03-26T21:10:39Z"
 content="""
What kind of encrypted remote are you using? An encrypted rsync special remote does not include the git repository, but only the content of the files, so cannot be used in this way.

If you set up a [[gcrypt special remote|special_remotes/gcrypt]], it will be encrypted and includes the full git repository, as well as the content of the files, so can be used like this.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkvN0eFnwvgw2JyTSSHw0QouytcxtPLln8"
 nickname="Stein Roald"
 subject="Need some clarification"
 date="2014-03-28T22:49:14Z"
 content="""
OK, thank you Joey for your help, and I think you have made some really excellent software. I've spent a couple of days to configure gpg and understand how to use it (glad I did, it has been on my todo-list for a long time...)

First to your question: When I started, I just used the git-annex assistant to \"set up a repository on a remote server using ssh\" as a \"transfer repository\". Now I'll use git-remote-gcrypt instead.

Now I have 2 questions:

---

1) On this webpage: [[https://github.com/joeyh/git-remote-gcrypt]] these are the instructions for setting up a remote for two participants:

    git remote add cryptremote gcrypt::rsync://example.com:repo
    git config remote.cryptremote.gcrypt-participants \"KEY1 KEY2\"
    git push cryptremote master

**Question 1:** As I own both computers myself, will syncing between the 2 computers work if I only use 1 KEY (the same KEY on both computers)?

---

2) I am still struggling to understand git-annex assistant and the use of special remotes. Important reasons for using git-annex are backups and moving data to the cloud to free up disk space locally. If I've understood it correctly, git-annex assistant always encrypts data sent to special remotes. And it seems to me that this page [[http://git-annex.branchable.com/special_remotes/]] suggest that git-annex assistant makes it easy to transfer files between different computers that do not communicate directly. I must clearly have misunderstood something, as Joey's comment above seem to state that I can't use the assistant for setting up syncing between computers.

**Question 2:** If I loose my computer, or it becomes corrupted, how can I get my files back if I don't set up special remotes as a gcrypt special remote? (And if I can't, what is the purpose of the special remotes made by the assistant?)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmNu4V5fvpLlBhaCUfXXOB0MI5NXwh8SkU"
 nickname="Adam"
 subject="Confused"
 date="2014-03-31T03:18:41Z"
 content="""
>What kind of encrypted remote are you using? An encrypted rsync special remote does not include the git repository, but only the content of the files, so cannot be used in this way.

Forgive me, Joey, I've been following your work on the assistant for almost as long as you've been working on it, and I use git a little bit, but I have yet to wrap my head around the complexities of git and how git-annex and the assistant...morph them.  I've read all the design docs you've written, but I still don't understand this.  From reading the OP it sounds like this is exactly what the encrypted rsync special remote is for: using a transfer repo to sync two devices that don't connect directly to each other.  Why would he need to use a gcrypt repo instead?

Thanks for your patience.  I know you get asked these basic questions over and over again.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.244"
 subject="comment 6"
 date="2014-04-02T20:11:35Z"
 content="""
@Adam, an encrypted special remote does not contain the git repository (the gcrypt special remote is an exception to this rule). So you need to use another method to sync the git repository between machines.

@Stein Roald once you have a clone of the git repository, you have the information that git-annex needs in able to get files from encrypted special remotes.

Question 1: While gcrypt supports multi-key setups, when you use the git-annex assistant to set up a gcrypt remote it only sets it up to use one key. It's left to you to arrange for this key to be on every computer that needs it. Or use command-line stuff to add additional keys later.

Question 2: You can certianly use the assistant to set up syncing between computers, but the only setup that currently provides 100% end-to-end enctyption of the git repository is using gcrypt. (Special remotes are 100% end-to-end encrypted, but as I've stated several times, do not contain the git repository data.)
Since you seemed to want 100% end to end encryption I suggested using gcrypt. There are simpler setups, like using XMPP, that encrypt everything but not end-to-end, so the XMPP server could snoop on it.

If you loose your computer, you can get your files back from any other device where you've set up a clone of that repository. A backup drive, another computer , etc.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkvN0eFnwvgw2JyTSSHw0QouytcxtPLln8"
 nickname="Stein Roald"
 subject="comment 7"
 date="2014-04-03T21:55:14Z"
 content="""
Thanks again, Joey, for your time and for your kind answers.

I have tried to set up git-annex with 2 laptops, each with their own gpg-key in order to use a remote server with gcrypt (and those laptops have shared the keys with each other). I regularly use git, but so far I haven't been able to get this setup to work. But before I bother the world with these problems, I would be interested in learning how to do the following:

*Question 3*: How can I use the git-annex assistant to set up a gcrypt remote (with one key)? It doesn't show up as an option as far as I can see (git-annex version: 4.20130815).

*Promise*: When I get git-annex to work with gcrypt, I'll make a summary on how I did it (and maybe which mistakes I made so others can avoid them).
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="216.145.95.162"
 subject="comment 8"
 date="2014-05-19T16:48:09Z"
 content="""
4.20130815 is too old. Get a current version.
"""]]


code:
Hello,

I went through several of the questions on the forum but could not find the answer to this. I have the following scenario, which seems very similar to the nomad case.

I have a server at home with many big media files. I want to keep a subset of these files on my laptop, to watch when I travel. When I'm done watching the files, I want to either delete them (i.e, they should no longer be on any machine) or archive them (i.e., they should remain on the server, but no longer be on the laptop). I think I know how to do this from the command line, but I still have a few questions.

- How do I drop a file from _every_ repository? If I understood correctly, `git annex drop foo` will only drop foo locally.
- Is it possible to follow this use case using the assistant, or should I use the command line instead (which would be very fine with me)? In particular, how do I choose the file I want to put on the laptop? (It seems `git annex get` is the command line way, is there an assistant equivalent?)
- Can I use both the assistant and the command line at the same time? Is there a description of the command-line version of what the assistant is doing behind the scenes?
- Can the server repository be both a "transfer" repository (to hold the files I haven't watch) and an "archive" repository (to keep the files I've watched and decided to keep), or do I need two repositories (on the server) for that?

Thanks a lot for any suggestion.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="64.134.31.139"
 subject="comment 1"
 date="2013-10-21T22:41:44Z"
 content="""
You can use `git annex drop foo --from remote` to drop the file from the remote. Of course, the remote has to be accessible. You can't drop files from an offline drive that's stored in a safe..

What you might be looking for is `git annex drop foo; rm foo`, followed by a git commit, and then later you can run `git annex unused` on a remote and it will let you then easily drop the files you've removed from the repository. This approach of just deleting files when you're done with them also works when using the assistant.

The assistant isn't really about only keeping a subset of files on your laptop. It would like to get them all, except for ones in \"archive\" directories.
You can configure the assistant to use manual mode, and then it doesn't download any files on its own (so you have to manually `git annex get` them), but it will still handle all the other stuff the assistant does, like automatically committing and syncing changes.

An archive repository wants one copy of every file that is not already stored in some other archive repository. So an archive repository could certainly be used instead of a transfer repository. (But not a small archive repository; those only want files that are moved to \"archive\" directories.) A better choice might be to make that server be a backup repository. That makes it want *every* file, no matter what, and it follows that it would archive everything, and have everything the client wants available for transferring to it.
"""]]


code:
[[!comment format=mdwn
 username="http://alan.petitepomme.net/"
 nickname="Alan Schmitt"
 subject="comment 2"
 date="2013-10-23T10:06:53Z"
 content="""
Thank you for the suggestions and explanations. I'm using a manual repo and I find that it works quite well.
"""]]


code:
Sorry, but all this wiki and the manpage seem to gloss over the most obvious question:

What happens when you commit conflicting edits in different repositories?


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="Good question!"
 date="2011-12-20T23:07:25Z"
 content="""
You get a regular git merge conflict, which can be resolved in any of the regular ways, except that conflicting files are just symlinks.

Example:

<pre>
$ git pull
...
Auto-merging myfile
CONFLICT (add/add): Merge conflict in myfile
Automatic merge failed; fix conflicts and then commit the result.
$ git status
# On branch master
# Your branch and 'origin/master' have diverged,
# and have 1 and 1 different commit(s) each, respectively.
#
# Unmerged paths:
#   (use \"git add/rm <file>...\" as appropriate to mark resolution)
#
#	both added:         myfile
#
no changes added to commit (use \"git add\" and/or \"git commit -a\")
$ git add myfile
$ git commit -m \"took local version of the conflicting file\"
</pre>
"""]]


code:
First of all, thanks for the amazing work! I've already tried Owncloud, SparkleShare, Unison, rsync and lsycnd. Finally I may have found the real deal.

My main problem with git-annex right now is that it is not fast/reliable enough. The main issues being: broken links now and then (I've probably done something wrong) and the slow: "startup scans", "consistency checks", "attempting to fix here" and "syncing with server". (50 Giga, Work-Server-Home, setup via webapp assistant)

Please, tell me if I'm wrong, but I have the impression that git-annex would get much more robust if it was 100% sure that nobody could mess with its file tree. One possibility would be to add the option (when creating a repo) to make a protected filesystem controlled by git-annex only via FUSE (not something like ShareBox). This could be a tiny filesystem (such as loggedfs) that does nothing but make sure that only git-annex can mount/change files. This would definitely add to speed (reducing checks) and stability.

I'm I being too naive? Most likely there is something I'm overlooking, like the amount of work this would involve.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.244"
 subject="comment 1"
 date="2014-04-05T22:34:32Z"
 content="""
git-annex can indeed be sped up by preventing modification of files in the tree. you can do this by running \"git annex indirect\"
"""]]


code:
[[!comment format=mdwn
 username="https://id.koumbit.net/anarcat"
 ip="2001:1928:1:9::1"
 subject="comment 2"
 date="2014-04-07T04:25:43Z"
 content="""
what's wrong with [sharebox](https://github.com/chmduquesne/sharebox-fs), actually?
"""]]


code:
[[!comment format=mdwn
 username="augusto"
 ip="177.98.104.136"
 subject="comment 3"
 date="2014-04-08T22:46:18Z"
 content="""
When I saw Sharebox's page on Github I had the impression it was vaporware. It has a section named \"Planned Interface\" and there are no updates for quite a while.

Is it working? How to install/use it?
"""]]


code:
Using the assistant, I have created a repository on my laptop plus a synced repository on a USB disk. Looking into the first repository, I see my files accompanied by a .git directory. However, looking on the USB disk (e.g. /media/usb/annex), all I see is what looks like the content of a .git directory.

This means that it is difficult to retrieve any file directly from this disk -- it has to be synced to another local repository first.

Is there any way to change this ? E.g. to have a copy of the working tree, plus a .git directory, on the disk ?

My use case: I have added plenty of media files to my repository. In addition to using the USB disk as a backup/medium for transfering these files to another computer, I'd like to be able to plug the disk to e.g. a media player and read the files directly from the tree, but it does not work at the moment.

Is there anything I am missing ?

Edited: yes, there is something I was missing: the forum entry at [[forum/USB_backup_with_files_visible/]]


code:
I have set up a remote server repository using the git-annex web assistant, accessed via ssh. The repository is a bare one according to the config file in the annex directory on the server.

I am wondering how I could access any of the files in the repository while logged in to the server - they don't have their usual file names to look for clearly. Is there a way to get a list of the files for starters? I tried using git annex find, but it never returns any files. git annex get followed by a filename of a document in the repository also doesn't work.

Thanks to help me understand better how to approach this.


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="173.162.44.162"
 subject="comment 10"
 date="2013-05-13T04:08:14Z"
 content="""
I'm not sure what's up.  I hope Joey can weigh in.

\"git annex sync\" ought to go get the symlinks from your remote, and merge them into your master branch, which is currently checked out, so you should see them.  :(
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 11"
 date="2013-05-13T17:33:26Z"
 content="""
When the assistant (or git annex sync) pushes to a repository, it creates a `synced/master` branch. So your bare repository has no regular `master` branch. So when you clone it, you get a repository with no branch checked out, which is what git-annex sync complains about.

There are several solutions. The easiest is to run `git merge origin/synced/master`; then your checkout will have a master branch and you can use `git annex sync` from then on.
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/bBy7WkgQicYHIiiyj.Vm0TcMbxi2quzbPFef#6f9f7"
 nickname="Frederik Vanrenterghem"
 subject="comment 12"
 date="2013-05-14T04:07:37Z"
 content="""
Thanks Joey, this worked. The symlinks are in the directory and I can get the actual file contents (except for one attempt that failed - I'll need to check why that didn't seem to have propagated to the bare repository.)
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 13"
 date="2013-06-12T18:56:39Z"
 content="""
Regarding my comment above, I've changed things so now `git annex sync` to a bare repository *does* push to the regular branch. So `git clone` will work without this bother.
"""]]


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="173.162.44.162"
 subject="comment 1"
 date="2013-05-08T13:40:16Z"
 content="""
Do you have git-annex available on the server from the command line?

If so, the easiest way would probably be to clone the repo in git, and do a \"git annex init\" in it.  This would give you a copy of the repo with none of the files in it.  It would be tiny, just full of symlinks, no content.  Then pull down copies of *only* the individual files you need using \"git annex get.\"

Maybe set that repo to \"untrusted\" to make sure other repos don't start depending on it to store copies of files to fulfill numcopies.

When you're done with a copy of a file, just \"git annex drop\" it again so it disappears from this skeletal repo.

"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/bBy7WkgQicYHIiiyj.Vm0TcMbxi2quzbPFef#6f9f7"
 nickname="Frederik Vanrenterghem"
 subject="comment 2"
 date="2013-05-10T03:21:57Z"
 content="""
I have git-annex on the VPS. The bare repository is in the directory \"annex\"

From the user's root directory, I execute:

git clone annex/ Documents
Cloning into 'Documents'...
done.
warning: remote HEAD refers to nonexistent ref, unable to checkout.

cd Documents/
~/Documents$ git annex init 'Documents on VPS'
init Documents on VPS ok
(Recording state in git...)

Still, when I try to get a file now, or do a git annex find, nothing shows up.

"""]]


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="99.54.57.201"
 subject="comment 3"
 date="2013-05-10T04:11:22Z"
 content="""
OK, I'm not sure what is up, but perhaps try \"git annex sync\" and see if that gives you the treeful of symlinks that you want.
"""]]


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="99.54.57.201"
 subject="comment 4"
 date="2013-05-10T04:11:53Z"
 content="""
that is, do \"git annex sync\" from within your new clone.
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/bBy7WkgQicYHIiiyj.Vm0TcMbxi2quzbPFef#6f9f7"
 nickname="Frederik Vanrenterghem"
 subject="comment 5"
 date="2013-05-10T04:23:28Z"
 content="""
Thanks for your help. Unfortunately that doesn't do the trick either:

git annex sync
(merging origin/git-annex into git-annex...)
(Recording state in git...)
commit
ok
git-annex: no branch is checked out

After this, git annex find still yields no results.
"""]]


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="173.162.44.162"
 subject="comment 6"
 date="2013-05-10T12:52:48Z"
 content="""
OK, I'm going to guess the problem is there is no \"master\" branch in your cloned repository, because there is no \"master\" branch in a bare remote. (Not sure about this, just guessing.)  Try \"git branch\" in your clone.  If there is no \"master\" branch, that's the problem.

If that's true, the right thing to do would probably have been not to clone from it, but to create a fresh git repo and then set your bare repo as a remote.

You could try doing that -- create a fresh git repo, git annex init it, and then try something like \"git remote add barerepo /path/to/bare/repo\" and then \"git annex sync\" which should pull changes from the bare repo's 'synced/master' branch into your 'synced/master' branch and from there into your master branch, giving you all your symlinks.

If that works you could declare your previous attempt \"dead\" and you'll be good.

It *may* also be possible to just fix the clone you've got by creating a master branch, maybe by branching off of synced/master.  Do a \"git branch\" and see whether synced/master exists, and just try \"git branch master synced/master\" and then \"git checkout master\" and that might do the job!

But I'm just throwing ideas around here, so I hope somebody who actually knows what's going on (joey? are you there?) will pipe up.
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/bBy7WkgQicYHIiiyj.Vm0TcMbxi2quzbPFef#6f9f7"
 nickname="Frederik Vanrenterghem"
 subject="comment 7"
 date="2013-05-12T14:04:29Z"
 content="""
That all sounded very plausible, but for some reason it failed to work for me. I went back to basics and tried getting there by replicating steps 1 to 3 of the walkthrough, but also to no aval. I'll still try one more thing - setting up a local repository on the VPS and syncing it to my main one over XMPP. That seems rather convoluted given the bare repository is on the same server, but who knows...
"""]]


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="173.162.44.162"
 subject="comment 8"
 date="2013-05-12T19:06:14Z"
 content="""
So what *does* show up?  Is your repo an empty directory?  Or are the symlinks that should point to content there?

as far as git goes, are you on the master branch?  *is* there a master branch?  If you do \"git status\" what do you see?

There's no reason this shouldn't work for you....
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/bBy7WkgQicYHIiiyj.Vm0TcMbxi2quzbPFef#6f9f7"
 nickname="Frederik Vanrenterghem"
 subject="comment 9"
 date="2013-05-12T23:51:16Z"
 content="""
I get the following:

frederik@niihau:~/Documents$ git status
# On branch master
#
# Initial commit
#
nothing to commit (create/copy files and use \"git add\" to track)

And this is the config:

frederik@niihau:~/Documents$ cat .git/config
[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
[remote \"origin\"]
        fetch = +refs/heads/*:refs/remotes/origin/*
        url = /home/frederik/annex
        annex-uuid = 1648a298-27aa-4cb1-xx-xxx
[annex]
        uuid = 4406bd35-9ff7-4008-yy-yyy
        version = 3
[remote \"niihau\"]
        url = /home/frederik/annex
        fetch = +refs/heads/*:refs/remotes/niihau/*
        annex-uuid = 1648a298-27aa-4cb1-xx-xxx

Not having worked with git before, all this is still very new for me. I have 33M of data in the .git directory, mostly in objects, so it does seem to have synced something already.
"""]]


code:
Hi!

One of my annex repository has a very strange behavior. Every `git annex` command is very slow.

I'm using MacOSX Mavericks, and this repository is a network drive, mounted with Samba. It's using direct mode, and the filesystem is crippled. I use *annex* for storing huge files, for example movies. I moved some files in this directory, and used `git annex add`. It was long (as checksum was performed) and I thought that everything was OK. I tried `git log -p`, it was OK too:

    new file mode 120000
    index 0000000..e58c65a
    --- /dev/null
    +++ b/Movies/movie.mp4
    @@ -0,0 +1 @@
    +../.git/annex/objects/FK/60/SHA256E-s346858581--053dca6a842376ab8022722df306ad5
    \ No newline at end of file

However it was not. I tried to launch `git annex sync another_repo` (with *another_repo* indirect and on a local disk) and it took ages. Even `git annex list` takes ages, **on every repository linked to this one**. With `ps -A`, I found out that the issue was created by `git --git-dir=/Volumes/SAMBA_REMOTE/.git --work-tree=/Volumes/SAMBA_REMOTE -c core.bare=false checkout -q -B annex/direct/master`.

**Have you ever noticed this behavior? Have I done something wrong?**

Here is the output of `git annex version`:

    git-annex version: 5.20131117-gbd514dc
    build flags: Assistant Webapp Pairing Testsuite S3 WebDAV FsEvents XMPP DNS Feeds Quvi TDFA CryptoHash
    key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
    remote types: git gcrypt S3 bup directory rsync web webdav glacier hook
    local repository version: 4
    default repository version: 3
    supported repository versions: 3 5
    upgrade supported from repository versions: 0 1 2 4


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.64"
 subject="comment 10"
 date="2013-11-26T21:48:11Z"
 content="""
Hmm. Seems to me that `git commit` is trying to download the whole big file from the SMB share. Perhaps just to compare it with the symlink it expects to be there?

When I try this, in a large direct mode repository with some video from *my* family, `git commit` does not open the file (verified with strace), let alone read it, and so finishes in well under 1 second.

Aha, I was able to reproduce `git commit` doing that with a repo on a FAT filesystem. git opens the file, and mmaps it, and I guess it proceeds to try to diff it against the symlink standin file it expected to be there.

This is particularly unfortunate, since `git commit` is only, I think, doing this so that it can print out a \"Changes not staged for commit\" message. Which git-annex sync throws away. There seems to be no git commit option that disables this behavior.

I think that this calls for making `git annex sync` not use `git commit` any longer, and instead manually build the commit using `git-write-tree` and `git-commit-tree`. Since I don't know if I will get to this before the thanksgiving holiday, I am creating a bug: [[bugs/direct_mode_sync_should_avoid_git_commit]]
"""]]


code:
[[!comment format=mdwn
 username="TroisSinges"
 ip="82.227.207.5"
 subject="comment 11"
 date="2013-11-26T21:57:52Z"
 content="""
Awesome! Thanks for your very efficient investigations. I'll stay tuned to the bug report.

Happy Thanksgiving!
"""]]


code:
[[!comment format=mdwn
 username="TroisSinges"
 ip="82.227.207.5"
 subject="comment 12"
 date="2013-11-27T12:08:29Z"
 content="""
I corrected the bug address: See [[/bugs/direct_mode_sync_should_avoid_git_commit/]]
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.64"
 subject="comment 13"
 date="2013-11-27T12:13:04Z"
 content="""
thanks!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.64"
 subject="comment 14"
 date="2013-12-02T20:05:00Z"
 content="""
Should be fixed; the git commit is now avoided.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.64"
 subject="comment 1"
 date="2013-11-26T17:26:12Z"
 content="""
If you run git-annex with --debug, it will print out every command it is running. This is useful because A) I can see the commands in context and B) it includes timestamps, which is a little bit more informative than \"it took ages\". (Which can mean anything.)

Anyway, I figured out the problem. Upgrade from v4 to v5 does what should be a one-time git checkout, but it seems that the auto upgrade code neglected to update annex.version, so it started doing it on every command run in a v4 repo. Fixed in git. You can work around the bug by running \"git config annex.version 5\".
"""]]


code:
[[!comment format=mdwn
 username="TroisSinges"
 ip="82.227.207.5"
 subject="comment 2"
 date="2013-11-26T20:42:03Z"
 content="""
My problem isn't solved. I cloned the repository from my netbook to the SMD drive, and added a file in this new direct annex repository :

    [3singes:/Volumes/Video/Videos/Films] $ git annex add MyFamily.mkv --debug
    [2013-11-26 19:52:59 CET] read: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"ls-files\",\"--others\",\"--exclude-standard\",\"-z\",\"--\",\"MyFamily.mkv\"]
    [2013-11-26 19:52:59 CET] read: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"ls-files\",\"--modified\",\"-z\",\"--\",\"MyFamily.mkv\"]
    [2013-11-26 19:52:59 CET] chat: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"cat-file\",\"--batch\"]
    add MyFamily.mkv [2013-11-26 19:52:59 CET] chat: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"check-attr\",\"-z\",\"--stdin\",\"annex.backend\",\"annex.numcopies\",\"--\"]
    (checksum...) [2013-11-26 19:52:59 CET] read: gsha256sum [\"MyFamily.mkv\"]
    [2013-11-26 20:55:35 CET] chat: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"cat-file\",\"--batch\"]
    [2013-11-26 20:55:35 CET] chat: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"hash-object\",\"-t\",\"blob\",\"-w\",\"--stdin\",\"--no-filters\"]
    ok
    (Recording state in git...)
    [2013-11-26 20:55:35 CET] feed: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"update-index\",\"-z\",\"--index-info\"]
    [2013-11-26 20:55:35 CET] chat: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"hash-object\",\"-w\",\"--stdin-paths\",\"--no-filters\"]
    [2013-11-26 20:55:35 CET] feed: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"update-index\",\"-z\",\"--index-info\"]
    [2013-11-26 20:55:35 CET] read: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
    [2013-11-26 20:55:36 CET] read: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"write-tree\"]
    [2013-11-26 20:55:36 CET] chat: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"commit-tree\",\"9bdb7b18b44020441ec8dc179e676120e769ff32\",\"-p\",\"refs/heads/git-annex\"]
    [2013-11-26 20:55:37 CET] call: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"update-ref\",\"refs/heads/git-annex\",\"293332232ca4e7428090171b52dcc5b3b80d1df6\"]

Let's try to sync to my macbookpro annex repository:

    [3singes:/Volumes/Video/Videos/Films] $ git annex sync macbookpro --debug
    [2013-11-26 20:57:48 CET] read: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"show-ref\",\"git-annex\"]
    [2013-11-26 20:57:48 CET] read: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
    [2013-11-26 20:57:49 CET] read: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"log\",\"refs/heads/git-annex..293332232ca4e7428090171b52dcc5b3b80d1df6\",\"--oneline\",\"-n1\"]
    [2013-11-26 20:57:49 CET] read: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"log\",\"refs/heads/git-annex..b2146cf0abc7d6a7e5c79d3b133181b56cd77461\",\"--oneline\",\"-n1\"]
    [2013-11-26 20:57:49 CET] read: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"log\",\"refs/heads/git-annex..c2cf7488b4745d4168491d8e62f5bdc354869133\",\"--oneline\",\"-n1\"]
    [2013-11-26 20:57:49 CET] chat: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"cat-file\",\"--batch\"]
    [2013-11-26 20:57:49 CET] read: git [\"config\",\"--null\",\"--list\"]
    [2013-11-26 20:57:49 CET] read: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"ls-files\",\"--stage\",\"-z\",\"--others\",\"--exclude-standard\",\"--\",\"/Volumes/Video/Videos\"]
    [2013-11-26 20:57:50 CET] chat: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"cat-file\",\"--batch\"]
    (Recording state in git...)
    [2013-11-26 20:57:55 CET] feed: xargs [\"-0\",\"git\",\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"add\",\"-f\"]
    commit
    [2013-11-26 20:57:55 CET] read: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"commit\",\"-m\",\"git-annex automatic sync\"]

Now it's 21:41 CET, and it's been stuck for circa 1 hour.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.64"
 subject="comment 3"
 date="2013-11-26T20:51:59Z"
 content="""
The problem you reported *is* solved. You were not talking about syncing before.

If your setup is such that `git commit` takes a long time to run, then `git annex sync` is also necessarily going to take a long time to run.
"""]]


code:
[[!comment format=mdwn
 username="TroisSinges"
 ip="82.227.207.5"
 subject="comment 4"
 date="2013-11-26T20:55:39Z"
 content="""
Thanks for your answer. On the second part of my problem, *why* should `git commit` be slow? It's only committing a file with a line (../.git/annex/...), isn't it?
"""]]


code:
[[!comment format=mdwn
 username="TroisSinges"
 ip="82.227.207.5"
 subject="comment 5"
 date="2013-11-26T21:02:17Z"
 content="""
Err, I read my post again, and I definitely *was* talking about syncing ;-)
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.64"
 subject="comment 6"
 date="2013-11-26T21:04:44Z"
 content="""
I think the most likely reason for git commit to be slow on your setup is that it probably rewrites .git/index. If you have a lot of files in your repository, the index file will be large and rewriting the index file will involve re-transferring it all over the network to the SMB share.

It's also possible that git commit scans the whole work tree, although I don't think it should -- it's not been told to with -a.

You may be able to find what's taking a long time by

    strace git --git-dir=/Volumes/Video/Videos/.git --work-tree=/Volumes/Video/Videos -c core.bare=false commit -m \"git-annex automatic sync\"

(or ltrace)
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.64"
 subject="comment 7"
 date="2013-11-26T21:07:12Z"
 content="""
If you kept your git repository on your local disk, and used a [[directory special remote|special_remotes/directory]] on the SMB share, you could then `git annex copy --to smbshare` and `git annex get --from smbshare` as desired, which would probably be much more efficient.
"""]]


code:
[[!comment format=mdwn
 username="TroisSinges"
 ip="82.227.207.5"
 subject="comment 8"
 date="2013-11-26T21:17:38Z"
 content="""
Thanks again for your answers.

`.git/index` is only 30Ko big. It shouldn't be an issue, even through the network.

However, I ran `lsof`, and strangely it lock the big file I added before (`/Volumes/Video/Videos/Films/MyFamily.mkv`)

    COMMAND   PID    USER   FD   TYPE DEVICE    SIZE/OFF                 NODE NAME
    git     68047 3singes  cwd    DIR   46,7       16384  5152641528667254045 /Volumes/Video/Videos
    git     68047 3singes  txt    REG    1,4     3035984             77501447 /Applications/git-annex.app/Contents/MacOS/bundle/git
    git     68047 3singes  txt    REG   46,7        8632  9219631544499731818 /Volumes/Video/Videos/.git/objects/pack/pack-99901c82bacfa26c04f505ceb27c4af9ccda414f.idx
    git     68047 3singes  txt    REG    1,4      403392             77501680 /Applications/git-annex.app/Contents/MacOS/bundle/usr/lib/libpcre.0.dylib
    git     68047 3singes  txt    REG   46,7        4012  8204150806531008435 /Volumes/Video/Videos/.git/objects/pack/pack-bb60a2315b13bda720682a4213507e5ee9ba029f.idx
    git     68047 3singes  txt    REG   46,7       15969 12003107822997399035 /Volumes/Video/Videos/.git/objects/pack/pack-bb60a2315b13bda720682a4213507e5ee9ba029f.pack
    git     68047 3singes  txt    REG    1,4      169392             77501683 /Applications/git-annex.app/Contents/MacOS/bundle/usr/lib/libz.1.dylib
    git     68047 3singes  txt    REG    1,4     2088992             77501673 /Applications/git-annex.app/Contents/MacOS/bundle/usr/lib/libiconv.2.dylib
    git     68047 3singes  txt    REG    1,4      600832             72809225 /usr/lib/dyld
    git     68047 3singes  txt    REG    1,4   342937194             75930310 /private/var/db/dyld/dyld_shared_cache_x86_64
    git     68047 3singes    0u   CHR  16,10    0t718314                 2413 /dev/ttys010
    git     68047 3singes    1w   CHR    3,2         0t0                  308 /dev/null
    git     68047 3singes    2w   CHR    3,2         0t0                  308 /dev/null
    git     68047 3singes    3u   REG   46,7           0 17793593705208135594 /Volumes/Video/Videos/.git/index.lock
    git     68047 3singes    4r   REG   46,7 15581987050 12221620371912261305 /Volumes/Video/Videos/Films/MyFamily.mkv
    git     68047 3singes    8w   CHR    3,2         0t0                  308 /dev/null

I'm using MacOSX, so I can only use `dtrace`, which I don't know so much.

**The process is over, after 1H30:**

    ok
    [2013-11-26 22:13:48 CET] read: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"symbolic-ref\",\"HEAD\"]
    [2013-11-26 22:13:48 CET] read: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"show-ref\",\"refs/heads/annex/direct/master\"]
    [2013-11-26 22:13:48 CET] call: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"show-ref\",\"--verify\",\"-q\",\"refs/heads/synced/master\"]
    [2013-11-26 22:13:49 CET] read: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"log\",\"refs/heads/annex/direct/master..refs/heads/synced/master\",\"--oneline\",\"-n1\"]
    pull macbookpro
    [2013-11-26 22:13:49 CET] call: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"fetch\",\"macbookpro\"]
    From /Users/3singes/Movies/NE PAS SAUVEGARDER/Videos
       b2146cf..c2cf748  git-annex  -> macbookpro/git-annex
    [2013-11-26 22:13:50 CET] call: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"show-ref\",\"--verify\",\"-q\",\"refs/remotes/macbookpro/annex/direct/master\"]
    [2013-11-26 22:13:50 CET] call: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"show-ref\",\"--verify\",\"-q\",\"refs/remotes/macbookpro/synced/master\"]
    [2013-11-26 22:13:50 CET] read: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"log\",\"refs/heads/synced/master..refs/remotes/macbookpro/synced/master\",\"--oneline\",\"-n1\"]
    ok
    [2013-11-26 22:13:50 CET] read: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"show-ref\",\"git-annex\"]
    [2013-11-26 22:13:50 CET] read: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
    [2013-11-26 22:13:50 CET] read: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"log\",\"refs/heads/git-annex..293332232ca4e7428090171b52dcc5b3b80d1df6\",\"--oneline\",\"-n1\"]
    [2013-11-26 22:13:51 CET] read: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"log\",\"refs/heads/git-annex..b2146cf0abc7d6a7e5c79d3b133181b56cd77461\",\"--oneline\",\"-n1\"]
    [2013-11-26 22:13:51 CET] read: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"log\",\"refs/heads/git-annex..c2cf7488b4745d4168491d8e62f5bdc354869133\",\"--oneline\",\"-n1\"]
    [2013-11-26 22:13:51 CET] call: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"branch\",\"-f\",\"synced/master\"]
    [2013-11-26 22:13:51 CET] call: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"branch\",\"-f\",\"master\"]
    [2013-11-26 22:13:51 CET] call: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"show-ref\",\"--verify\",\"-q\",\"refs/remotes/macbookpro/synced/master\"]
    [2013-11-26 22:13:51 CET] read: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"log\",\"refs/remotes/macbookpro/synced/master..refs/heads/synced/master\",\"--oneline\",\"-n1\"]
    [2013-11-26 22:13:51 CET] call: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"show-ref\",\"--verify\",\"-q\",\"refs/remotes/macbookpro/git-annex\"]
    [2013-11-26 22:13:51 CET] read: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"log\",\"refs/remotes/macbookpro/git-annex..git-annex\",\"--oneline\",\"-n1\"]
    push macbookpro
    [2013-11-26 22:13:52 CET] call: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"push\",\"macbookpro\",\"+git-annex:synced/git-annex\",\"annex/direct/master:synced/master\"]
    Counting objects: 9, done.
    Delta compression using up to 8 threads.
    Compressing objects: 100% (4/4), done.
    Writing objects: 100% (5/5), 440 bytes | 0 bytes/s, done.
    Total 5 (delta 3), reused 0 (delta 0)
    To /Users/3singes/Movies/NE PAS SAUVEGARDER/Videos
       c2cf748..2933322  git-annex -> synced/git-annex
    [2013-11-26 22:13:57 CET] read: git [\"--git-dir=/Volumes/Video/Videos/.git\",\"--work-tree=/Volumes/Video/Videos\",\"-c\",\"core.bare=false\",\"push\",\"macbookpro\",\"master\"]
    ok

"""]]


code:
[[!comment format=mdwn
 username="TroisSinges"
 ip="82.227.207.5"
 subject="comment 9"
 date="2013-11-26T21:19:42Z"
 content="""
I thought to a directory special remote, but it doesn't meet my needs: the SMB drive is used by a MediaCenter, so I need the working tree.
"""]]


code:
I have been playing around with annex assistent today (way late as I am one of the Kickstarter backers) and found the following puzzling.

I am running a fairly new version, my package manager calls it 4.20130323 (git-annex does not seem to have a --version switch).

I have been playing around with the annex assistent webapp today and have the following questions/observations:

1) The "remote" server creates a bare git repository as far as I can tell, the "local computer" probably does the right thing (but I cannot run it as the "other box" has no X or browser). Is there any way to remotely create a "local" copy (i.e. non-bare git repository, which can later be managed by the cli?)
2) It seems like git-annex-assistent (webapp) binds to localhost, is it possible to let it bind to either a specific ip-address (or all)? (Yes, I understand the security implications, but the use-case is the box from question 2 - i.e. on local network but no X and no browser).
3) what is being launched when I hit "files", on the video it starts a file-manager on my box nothing happens (and no errors as far as I can tell).

Thank you in advance

Svenne


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-04-08T19:11:33Z"
 content="""
1. If you want to use the cli for managing the repository, why not `git clone` it using the cli as well? Set up a remote in your existing repository pointing at the repository you cloned, and the assistant will push changes to it. This is all that the webapp would do when creating the repository. The webapp does not set up remote non-bare repositories because unless the user is comfortable with the cli, nothing is going to keep them up-to-date.

2. Setting up the assistant on a headless box is something I need to support better, indeed. I've just now made `git annex webapp --listen=IP` listen on the specified IP address, and output the URL you can connect to from the remote side. Note that this does not yet use HTTPS, so use with caution.

3. It uses the `xdg-open` program to open a file manager. If `xdg-open` was not in your PATH, it would instead point the web browser at a file:/// URI as a fallback. If it seems to do nothing, this probably means `xdg-open` is in PATH but not working for some reason. Try running `xdg-open /path/to/your/repository`
"""]]


code:
Hello,

I am just starting to learn git-annex so forgive me if this is a naive question.

I have the following repositories:

1. Home (laptop)
2. Work (mac mini)
3. S3 bucket
4. USB drive, full backup, attached to Home laptop.

I want to sync files between Home and Work via the S3 bucket.  It is not clear to me how to accomplish this through the git-annex assistant.  Is this possible? Are there instructions online?

Thanks in advance!

Scott


code:
[[!comment format=mdwn
 username="http://marco.paga.myopenid.com/"
 ip="2001:4dd0:ff00:917b:d575:780:2c27:d982"
 subject="Yes, it is possible"
 date="2013-08-19T17:59:56Z"
 content="""
You can sync both repos. The simplest way is to have both clients signed on to jabber. This way the clients exchange information when the two are online.

Additionally you need to have a cloud special remote available on both ends. The cloud repository is needed to push and retrieve data for the clients. You need to configure the cloud as a transfer reposity. The clients as - what else - client. The USB Drive can be used as a backup repository on one or both ends and perhaps even for syncing very large files around.
"""]]


code:
[[!comment format=mdwn
 username="http://marco.paga.myopenid.com/"
 ip="2001:4dd0:ff00:917b:d575:780:2c27:d982"
 subject="I forgot to mention"
 date="2013-08-19T18:01:47Z"
 content="""
You might want to have a look at this: https://git-annex.branchable.com/assistant/remote_sharing_walkthrough/
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkjQhXk8KAh9yD0p1R6QzT-Sw7FtHE3d54"
 nickname="Scott"
 subject="comment 3"
 date="2013-08-19T18:03:41Z"
 content="""
Great thanks!  I'll try it out tonight and report back.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkjQhXk8KAh9yD0p1R6QzT-Sw7FtHE3d54"
 nickname="Scott"
 subject="comment 4"
 date="2013-08-20T16:18:24Z"
 content="""
When I try to setup my Jabber client, using my gmail address, the assistant crashes.  I thought this might be due to having two-factor authentication enabled so I generated an application specific password and tried that.  The assistant still crashes.  The log is huge, I am not sure what would be relevant to show here.
"""]]


code:
[[!comment format=mdwn
 username="http://marco.paga.myopenid.com/"
 ip="2001:4dd0:ff00:917b:f87b:5579:3820:5db"
 subject="Please post the log"
 date="2013-08-21T15:49:00Z"
 content="""
I don't know what is going wrong. It is not a bad idea to post the log information.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkjQhXk8KAh9yD0p1R6QzT-Sw7FtHE3d54"
 nickname="Scott"
 subject="comment 6"
 date="2013-08-21T20:05:31Z"
 content="""
I tried a different approach and attempted to setup s3 at the command line.  This is what I did:

zombie:annex scott$ export AWS_ACCESS_KEY_ID=\"my key\"
zombie:annex scott$ export AWS_SECRET_ACCESS_KEY=\"my secret key\"
zombie:annex scott$ git annex initremote S3 type=S3 encryption=shared
[2013-08-21 13:03:42 PDT] read: git [\"--git-dir=/Network/Servers/filer004/vol/office_homes/scott/annex/.git\",\"--work-tree=/Network/Servers/filer004/vol/office_homes/scott/annex\",\"show-ref\",\"git-annex\"]
[2013-08-21 13:03:42 PDT] read: git [\"--git-dir=/Network/Servers/filer004/vol/office_homes/scott/annex/.git\",\"--work-tree=/Network/Servers/filer004/vol/office_homes/scott/annex\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
[2013-08-21 13:03:42 PDT] read: git [\"--git-dir=/Network/Servers/filer004/vol/office_homes/scott/annex/.git\",\"--work-tree=/Network/Servers/filer004/vol/office_homes/scott/annex\",\"log\",\"refs/heads/git-annex..da801570f9ed8d28e5a0cea6cc51f1a2003317d6\",\"--oneline\",\"-n1\"]
[2013-08-21 13:03:42 PDT] read: git [\"--git-dir=/Network/Servers/filer004/vol/office_homes/scott/annex/.git\",\"--work-tree=/Network/Servers/filer004/vol/office_homes/scott/annex\",\"log\",\"refs/heads/git-annex..8015f3fed32792b558d16008d20816fab0fc50c2\",\"--oneline\",\"-n1\"]
[2013-08-21 13:03:42 PDT] chat: git [\"--git-dir=/Network/Servers/filer004/vol/office_homes/scott/annex/.git\",\"--work-tree=/Network/Servers/filer004/vol/office_homes/scott/annex\",\"cat-file\",\"--batch\"]
[2013-08-21 13:03:42 PDT] read: git [\"config\",\"--null\",\"--list\"]
initremote S3 (encryption setup) [2013-08-21 13:03:42 PDT] read: gpg [\"--quiet\",\"--trust-model\",\"always\",\"--gen-random\",\"--armor\",\"2\",\"512\"]
(shared cipher) (checking bucket...)
git-annex: connect: does not exist (Connection refused)
failed
git-annex: initremote: 1 failed
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkjQhXk8KAh9yD0p1R6QzT-Sw7FtHE3d54"
 nickname="Scott"
 subject="comment 7"
 date="2013-08-21T20:07:58Z"
 content="""
Sorry that didnt format properly

    zombie:annex scott$ export AWS_ACCESS_KEY_ID=\"key\"
    zombie:annex scott$ export AWS_SECRET_ACCESS_KEY=\"secret\"
    zombie:annex scott$ git annex initremote S3 type=S3 encryption=shared
    [2013-08-21 13:03:42 PDT] read: git [\"--git-dir=/Network/Servers/filer004/vol/office_homes/scott/annex/.git\",\"--work-tree=/Network/Servers/filer004/vol/office_homes/scott/annex\",\"show-ref\",\"git-annex\"]
    [2013-08-21 13:03:42 PDT] read: git [\"--git-dir=/Network/Servers/filer004/vol/office_homes/scott/annex/.git\",\"--work-tree=/Network/Servers/filer004/vol/office_homes/scott/annex\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
    [2013-08-21 13:03:42 PDT] read: git [\"--git-dir=/Network/Servers/filer004/vol/office_homes/scott/annex/.git\",\"--work-tree=/Network/Servers/filer004/vol/office_homes/scott/annex\",\"log\",\"refs/heads/git-annex..da801570f9ed8d28e5a0cea6cc51f1a2003317d6\",\"--oneline\",\"-n1\"]
    [2013-08-21 13:03:42 PDT] read: git [\"--git-dir=/Network/Servers/filer004/vol/office_homes/scott/annex/.git\",\"--work-tree=/Network/Servers/filer004/vol/office_homes/scott/annex\",\"log\",\"refs/heads/git-annex..8015f3fed32792b558d16008d20816fab0fc50c2\",\"--oneline\",\"-n1\"  ]
    [2013-08-21 13:03:42 PDT] chat: git [\"--git-dir=/Network/Servers/filer004/vol/office_homes/scott/annex/.git\",\"--work-tree=/Network/Servers/filer004/vol/office_homes/scott/annex\",\"cat-file\",\"--batch\"]
    [2013-08-21 13:03:42 PDT] read: git [\"config\",\"--null\",\"--list\"]
    initremote S3 (encryption setup) [2013-08-21 13:03:42 PDT] read: gpg [\"--quiet\",\"--trust-model\",\"always\",\"--gen-random\",\"--armor\",\"2\",\"512\"]
    (shared cipher) (checking bucket...)
    git-annex: connect: does not exist (Connection refused)
    failed
    git-annex: initremote: 1 failed
"""]]


code:
I need some advice from more experienced users.

I've created four repositories on my macbook and paired them with repos on my ubuntu desktop. They are all created with the webapp so they might have some configurations that are not optimal for me.

In fact, when I start my macbook it is unusual for more than an hour because git-annex is reading and writing to and from the disk in the background.

I don't change much in three of the four repos, only one repo contains my daily work.

What configuration would enhance my situation?

The four repos are called:

- movies
- pictures
- music
- documents (many changes each day)

I have also problems to switch to the documents repo in the webapp.

TIA
juh


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmTIL7ubr5opWM69Q5VtCxuxC2H0SSnzic"
 nickname="Petr"
 subject="Where are your repositories located?"
 date="2014-01-14T08:58:31Z"
 content="""
First - Where are the repos actually located ~/Desktop or ~/annex/<repo>? Other?

I can confirm, that like two weeks back I have had similar issue using git-annex-assistant with other synced repos on local LAN and at Box.com. The git-annex processes actually overloaded the CPU and I was forced to kill annex processes to get over.

I also found myself that NFS mounted shares, linked by symlink to home directory, can slowdown mac finder app as well if NFS has slight delays in response.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnX1msQxnLoSeu7q-i-c9BWghonsN7Qmns"
 nickname="Jan Ulrich"
 subject="comment 2"
 date="2014-01-14T10:23:21Z"
 content="""
All repositories are located in my home folders:
~/Pictures
~/Movies
~/Music
~/Documents

On my Ubuntu desktop machine and my macbook. They are connected in my local network via WLAN.

I use different git annex versions on these two machines. Maybe this causes problems.

On Ubuntu I have
Version: 4.20130815
Build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP

On my macbook:
Version: 5-20140106-gcb3351b

juh
"""]]


code:
Hi,

Does the Android app support encryption at all? The assistant allows me to create encrypted repositories but it just keeps telling me to install git-remote-gcrypt, which did not work.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.55"
 subject="comment 1"
 date="2014-07-03T19:14:37Z"
 content="""
Encrypted [[special_remotes]] are supported just fine on Android.

As for storing the [[encrypted git repository itself on a remote|special_remotes/gcrypt]], which is what git-remote-gcrypt is used for, it would need porting that to Android. I don't think that should be very hard (it's just a rather complicated shell script and all the tools it uses are already included in the git-annex bundle). I may eventully end up reimplementing git-remote-gcrypt as part of git-annex in Haskell, since porting it to eg, Windows is likely to be a lot harder..
"""]]


code:
While running the Git Annex App on Android, the app causes a constant cpu usage of about 50% when idling. I've seen this behavior on two devices (phone and tablet) with a CM 10.1 nightly build. The app causes this high cpu usage even when it is in the background, not performing any synchronization and managing only one repository containing just one file. Unfortunately I couldn't figure out what causes the cpu usage. The daemon.log file remains unchanged and I couldn't find any other log files.

Is this expected behavior or unusual high cpu usage?


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmKKg3Vmzk7KwRGRKjHVdtyoj1JfxLX6NM"
 nickname="Tom"
 subject="comment 1"
 date="2013-10-01T17:38:03Z"
 content="""
I've had this issue as well. Saw a comment on Joey's blog that implies he knows about it and that a fix will be released soon.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="64.134.31.139"
 subject="comment 2"
 date="2013-10-16T16:57:02Z"
 content="""
This was fixed.

(Please file bug reports for bugs in the future, so they don't clutter up this forum once fixed.)
"""]]


code: Is there a good way to use a ext4 formated external sdcard on a rooted android with symlinks. If android mounts it (with the sdcard command i think) symlinks don't work. If i mount it with mount the permissions are a problem because every app has a different user.


code:
Hey!

I have got a Kindle Fire HD (2nd generation) and want to use Git Annex to sync my data round. However, it does not sync my data at all (my other setups, e.g. PC or laptop do work quite well). The reason for that seems to be quite simple, but I do not know how to fix that:

I always get the warning message:

> TransferScanner crashed: /storage/emulated/legacy/annex/.git/index: copyFile: does not exist (No such file or directory).

I found the same error in the log:

> git-annex: /storage/emulated/legacy/annex/.git/index: copyFile: does not exist (No such file or directory)
> ...
> fatal: Run with no arguments or with -c cmd
> git-annex-shell: git-shell failed

I am using the latest versions for android and arm (server, raspberry pi) from this site from today (13.07.2014)

http://git-annex.branchable.com/install/

I hope somebody can help me. I tried restaring the TransferScanner, restarting git-annex, restarting the OS, resetting the OS (to shipping defaults), nothing helped.

Cheers,
Stephan


code:
[[!comment format=mdwn
 username="sts"
 ip="134.147.119.104"
 subject="comment 1"
 date="2014-07-14T09:12:13Z"
 content="""
Ok, I checked and the index file was missing and not sure why, but I had to manually sync one time. So I had to go in the annex folder with the command line and make a \"git annex sync\". It works pretty well now, it seems not to automatically sync my data (have to force it with \"Sync Now\"), but at least I can sync data to my tablet and thats awesome :).
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 2"
 date="2014-07-14T18:21:15Z"
 content="""
How did you create this repository on the android device? Normally a git repository has an index file, unless it has just been manually created with `git init` and has not yet had anything added to it. When the webapp is used to create a repository, it should always leave it with an index file already present.
"""]]


code:
Joey,

I have git-annex now to manage many of the repositories on my system.  I have them both on my local machine, and on a very large file server, and a backup system on the Internet.

Today I went to look at a file in one of my annexes and it wasn't there.  This really surprised me.  But what surprised me most is that around 90% of the files in *all* of my annexes on both my local system and my file server are completely missing.  Only the Internet backup system has them.

How could something like this happen, when I haven't been interacting with these annexes at all during this time?  Can you think of any scenario that might lead to this?  This is pretty much the absolute worst case scenario for an archival data system.

I am running on Mac OS X 10.8, using GHC 7.6.3 to build git-annex, and I keep my git-annex binary updated often.

Thanks,
  John


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/2grhJvAC049fJnvALDXek.6MRZMTlg--#eec89"
 nickname="John"
 subject="comment 1"
 date="2013-08-23T03:06:00Z"
 content="""
Wait, I think this comes from a backend switch.  I changed my .gitattributes file at one point to read:

* annex.backend=SHA512E annex.numcopies=2

I thought this would just affect new files, not existing annexed content.  Could this do it?
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/2grhJvAC049fJnvALDXek.6MRZMTlg--#eec89"
 nickname="John"
 subject="comment 2"
 date="2013-08-23T03:14:15Z"
 content="""
Yes, that was the problem, sorry to bother you.  None of my data is gone, it's just sitting there under unknown names.  I can roll back to when it knew the names, and migrate them forward.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.63"
 subject="comment 3"
 date="2013-08-23T04:28:45Z"
 content="""
If you change the backend, and then in one repository you run `git annex migrate`, other repositories that have the old keys will not know about the new names. For this reason, then multiple repositories have the files, it's best to run it redundantly in each repository.

TBH, migration is a bit of a PITA because of this. Best to aovid it in most cases.

Git-annex will never perform a migration begind your back. You must have run `git annex migrate` at some point. You can check the git history for details.
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/2grhJvAC049fJnvALDXek.6MRZMTlg--#eec89"
 nickname="John"
 subject="comment 4"
 date="2013-08-30T06:06:16Z"
 content="""
Just to confirm, this wasn't a git-annex problem at all, but just a misstep during migration as you suggested.

I think what I'm going to do now is to just wipe the slate clean and start over again, by using `unannex -fast` on all the files, wiping `.git`, and then adding everything back in using my new default backend of SHA512E.  The bigger pain is doing the same thing on all the servers where I have this data (to avoid having to upload it again), but in such a way that I'm not replicating file history.  I think I should be able to just clone, `mv $OLDREPO/.git/annex/objects objects`, `git annex add objects`, `git rm -r --cached objects`, and then everything should be good without even needing a new commit on the remote machine, just a git-annex sync.
"""]]


code:
It seems like one of my repos dropped files by itself, my setup involved 3 distributed repos (non bare repos, not using assistant), A/B/C then one client died I've created another one in its place called D cloned from B and marked C as dead. I have now A/B/D. I have updated files in A synced it to D (central) I then tried dropping unused, it listed none (I've dropped the files from A which has full copy) at this point I thought B did not sync to it maybe thats why it did not drop them. I went home sync my laptop B (B dropped its share of unused files) but cental D still show no unused.

A-D has full copies of the repo
B has partial.

At this point I checked out a old commit from a week ago, checked where the links for the files are pointing to, they point to no where files are not there. Repo sized reported with du -hcs show both repos are the same size (132GB) looks like D managed to drop unused files by itself. I am wondering what caused this?

My annex version is,

git-annex version: 4.20130902-g307537a

on all clients.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.51"
 subject="comment 1"
 date="2013-09-12T21:01:03Z"
 content="""
If you have just created repository D and have just copied some files to it, then it will only contain those files, and not old deleted files or old versions of files. You can run `git annex copy --all --to D` to copy all those files to D, although if you're only wanting to dropunused them, there is not much point in that..
"""]]


code:
I've been using annex for some weeks now and while I more and more love how it behaves on my Linux machines, I just can't get it working on Windows...

The setup consists of two Ubuntu machines (one being an always-on server) and a Windows laptop I keep for the occasional moment of gaming. My wife's Windows machine is a candidate to join the annex setup, as well as some other computers I still use every now and then.

The first thing I started annexing was my pictures folder. It consists of about 40k files and occupies about 350GB. The Ubuntu server is running a hidden service as a substitute for a DynDNS with ssh basically the only thing going out/in. First, I couldn't get annex to properly sync its data via TOR, but when I instead set up a directory special remote on the server (with the annex repository there being "bare", i.e. not containing any actual data) I got them to sync as they should.

Not the Windows machine though. It's just too slow. It seems that on every sync, every add, every anything it scans... well, everything. I added some files and after an hour I checked the resource monitor which files were being accessed and it seemed to be every last one of the files present in the pictures folder. I'm not sure what's going on there, but it's really getting to the point of being a dealbreaker... I think it's something with annex only supporting direct mode repositories on Windows and all files just lying around (as that's really the only large difference between those setups), although I'm of course not entirely sure.

I'd really like to run annex in indirect mode on Windows. I can't really find any information on that (except those few "official" pages that just state that annex is running in direct mode on Windows). I know that creating symlinks needs elevated priviledges on Windows, but accepting a prompt seems to be much more realistic than waiting hours for a sync.

So... Is there any way to get indirect running on Windows? And if there isn't, is there any other way to speed up direct mode / Windows performance?


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnOSgFb3l7nL3Fs7Y9gPGJJjFiV7aJ1tek"
 nickname="Phil"
 subject="Same boat"
 date="2014-07-03T19:38:15Z"
 content="""
I'd love to know this too. I'm setting up on 2 Windows 8.1 machines (with a 3rd Windows 7 hopefully joining the party as well as a couple of Linux boxes) but doing anything is really slow. I just ran git annex status on a 50GB, 7000 file Music repo and it took 6 minutes. I've not even setup the remotes yet but as soon as I'd added and sync'd, things slowed down. I'm running on a nippy SSD and CPU usage is around 12%.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnOSgFb3l7nL3Fs7Y9gPGJJjFiV7aJ1tek"
 nickname="Phil"
 subject="comment 2"
 date="2014-07-03T19:39:12Z"
 content="""
Fortunately, git annex sync is pretty quick
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.55"
 subject="comment 3"
 date="2014-07-04T20:57:54Z"
 content="""
Yall need to tell me your git-annex versions. [[devblog/day_183__rubbing_sticks_together]] fixed a horrible slowdown that indeed causes git-annex on windows to unnecessarily look at lots of files.

Also, is this using the git-annex assistant, or using git-annex at the command line that's being slow?

As to using symlinks on windows, cygwin's build of git is able to handle them somehow. I have not tried to get it working with git-annex, but it seems at least possible that indirect mode could be accomplished that way.
"""]]


code:
[[!comment format=mdwn
 username="divB"
 ip="171.67.172.69"
 subject="Also in the same boat"
 date="2014-07-07T01:12:18Z"
 content="""
Even worse here. git-annex looked (and still looks) so promising for exactly I want (share >100GB photos/data with my girlfriend which are on my private DSL connected server but each of us should only download whatever he currently needs for offline usage and everything should stay consistent, e.g. is one edits, adds or deletes files). However, after long testing in Windows I gave up. I had so many issues and speed was one of them. So I looked again at other solutions (ownCloud pydio) but they are not good either. unison is one thing that just works perfectly but it is not suited to sync huge stuff where just parts are available on one machine.

I still put much hope in git-annex that it's useable at some point also in Windows. If there would be another of these kickstarters for that, I'd be in.

One BIG dealbreaker for me is SSH. It popped up a couple of times already there is inconsistency between ssh.exe, plink.exe and %GIT_SSH%.
But the biggest issue is that ssh.exe in Windows does not do connection sharing: For each single file a new SSH session is created! That way it works only for huge files (movies) where the overhead is small. In my tests I had smaller files, that's just unuseable.

I know it's a conundrum. unison just leaves one ssh connection open, that would be nice. I know however that this requires bigger changes. But what about for example starting something in background (during a git-annex command) that leaves the SSH session open and pumps data via some IPC?

Last but not least: links. I agree windows links are unuseable. But would it really be real links? What about just basic *.lnk files?

"""]]


code:
[[!comment format=mdwn
 username="divB"
 ip="171.67.172.69"
 subject="comment 5"
 date="2014-07-07T01:16:21Z"
 content="""
One more thing: integration in TortoiseGit would be sooooooooo great! You would then even be able to let novice computer users use git-annex.

I added a feature request long time ago but it's on hold: http://code.google.com/p/tortoisegit/issues/detail?id=2166

I do hope that git-annex becomes more popular to increase the chance that it will be built into that...

"""]]


code:
[[!comment format=mdwn
 username="Idhup"
 ip="178.175.139.138"
 subject="Connection sharing workaround"
 date="2014-07-07T10:20:23Z"
 content="""
I had the same issues but was able to work around most of them. I simply created keypairs on my Windows machines that allowed me to log in to my Ubuntu server. Means that while the overhead does make it smaller (I guess), the whole sync process can simply be run in the background without me having to check for a password prompt for 30 minutes, just to miss the short period during which I can enter it. Also, the actual object storage was moved to a directory special remote on the server, so Windows clients can simply automount a samba share and push the content over that line, which is less of a hazzle.

Also, the speed issue got *much* better when I added all files. I still had a lot of files just lying around on my Windows machine. Once I added them all, the sync process sped up by several magnitudes. So, if https://git-annex.branchable.com/forum/fatal:_Out_of_memory__63___mmap_failed:_No_such_file_or_directory/ can be solved, I guess I'm happy on Windows.
"""]]


code:
[[!comment format=mdwn
 username="Idhup"
 ip="178.175.139.138"
 subject="typo"
 date="2014-07-07T10:23:23Z"
 content="""
Slower. It's not smaller, it's slower. Gee.
"""]]


code:
I'm trying to use git-annex as a way to sync fairly large folders between machines. These folders contain documents, projects, ...

There are several issues with files that just won't get added to my git-annex repository:

- files mentioned in .gitignore are not added -- this is good.
- files starting with a dot are not added unless I explicitly mention them to git add -- this is strange, but I can work around it.
- folders called .git/ are not added no matter what I try. git annex add --force doesn't help, either.

Any idea how I could solve the third issue? It might not even be specific to git-annex... maybe git just works that way... but I'd be grateful for hints.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlq4ClC5EMN1Vq1DpWXAqP5TiDnCK1mSfk"
 nickname="Jonas"
 subject="comment 1"
 date="2014-01-09T22:58:30Z"
 content="""
I just found [this thread](http://git-annex.branchable.com/forum/Tracking_a_directory_with_some_hg_and_git_repositories/) where a user was discouraged from syncing git repositories with annex.

I've used unison in the past for this. Sure, it would occasionally create a few unneeded object files; but these got removed during the next \"git gc\", and never corrupted a repository... Do people know of any better way to handle this problem?
"""]]


code:
[[!comment format=mdwn
 username="https://launchpad.net/~timo-linux"
 nickname="Tim O'Callaghan"
 subject="How about automatically adding it as a submodule?"
 date="2014-01-11T10:30:57Z"
 content="""
Adding as a submodule and automatically adding the annexe destination as a remote? that should work in most cases..
"""]]


code:
I'm hoping to use git-annex in a project to manage large image files and maintain archived copies of them on removable USB drives. But I can't get this to work the way it's described in the walkthrough and in the "git-annex/tips/offline archive drives" example. Here's what I did:

    marshal@home[~]> git-annex version
    git-annex version: 5.20140412ubuntu1
    build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV Inotify DBus DesktopNotify XMPP DNS Feeds Quvi TDFA CryptoHash
    key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
    remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier hook external

Create the local repo:

    marshal@home[~]> mkdir ~/annex
    marshal@home[~]> cd ~/annex
    marshal@home[~/annex]> git init
    Initialized empty Git repository in /home/marshal/annex/.git/
    marshal@home[~/annex]> git annex init "my laptop"
    init my laptop ok
    (Recording state in git...)

Create the remote repo (assume the warning can be ignored?)

    marshal@home[~/annex]> cd /media/marshal/Sony\ USB/
    marshal@home[/media/marshal/Sony USB]> git clone ~/annex
    Cloning into 'annex'...
    done.
    warning: remote HEAD refers to nonexistent ref, unable to checkout.
    marshal@home[/media/marshal/Sony USB]> cd annex
    marshal@home[/media/marshal/Sony USB/annex]> git annex init "usbdrive"
    init usbdrive ok
    (Recording state in git...)

Make them remotes of each other

    marshal@home[/media/marshal/Sony USB/annex]> git remote add laptop ~/annex
    marshal@home[/media/marshal/Sony USB/annex]> cd ~/annex
    marshal@home[~/annex]> git remote add usbdrive /media/marshal/Sony\ USB/annex/

Add some files into the local repo

    marshal@home[~/annex]> cp ~/Desktop/file* .
    marshal@home[~/annex]> git annex add .
    add file1.m3u ok
    add file2.sh ok
    add file3.pdf ok
    (Recording state in git...)
    marshal@home[~/annex]> git commit -a -m "added files"
    [master (root-commit) 4ea3be7] added files
     3 files changed, 3 insertions(+)
     create mode 120000 file1.m3u
     create mode 120000 file2.sh
     create mode 120000 file3.pdf

Change the preferred content settings

    marshal@home[~/annex]> git annex group usbdrive archive
    group usbdrive ok
    (Recording state in git...)
    marshal@home[~/annex]> git annex wanted usbdrive standard
    wanted usbdrive ok
    (Recording state in git...)

Sync to the usbdrive repo

    marshal@home[~/annex]> git annex sync usbdrive
    commit  ok
    pull usbdrive
    remote: Counting objects: 7, done.
    remote: Compressing objects: 100% (3/3), done.
    remote: Total 5 (delta 1), reused 0 (delta 0)
    Unpacking objects: 100% (5/5), done.
    From /media/marshal/Sony USB/annex
     * [new branch]      git-annex  -> usbdrive/git-annex
    ok
    (merging usbdrive/git-annex into git-annex...)
    (Recording state in git...)
    push usbdrive
    Counting objects: 29, done.
    Delta compression using up to 8 threads.
    Compressing objects: 100% (22/22), done.
    Writing objects: 100% (25/25), 2.44 KiB | 0 bytes/s, done.
    Total 25 (delta 3), reused 0 (delta 0)
    To /media/marshal/Sony USB/annex/
     * [new branch]      git-annex -> synced/git-annex
     * [new branch]      master -> synced/master
    ok

Copy eveything to the usbdrive repo

    marshal@home[~/annex]> git-annex copy --auto --to usbdrive
    copy file1.m3u (to usbdrive...)
    SHA256E-s64061--c388e2c927cb4c88c11b1b5a8e166c4d8080dc528d53954dad1ee6d4933ce3c7.m3u
             64,061 100%   29.84MB/s    0:00:00 (xfr#1, to-chk=0/1)
    ok
    copy file2.sh (to usbdrive...)
    SHA256E-s318--b8729560cc759d2256903feaa4ba65994ccbac94f30515d52a39083ad52e1bad.sh
                318 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=0/1)
    ok
    copy file3.pdf (to usbdrive...)
    SHA256E-s32239--9b051067d5ef9f0cb4a01750901bf6bee3f9348e5b10138c8ef416ac8d51e5df.pdf
             32,239 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=0/1)
    ok
    (Recording state in git...)

Check to see where git-annex "thinks" the files are

    marshal@home[~/annex]> git-annex whereis file1.m3u
    whereis file1.m3u (2 copies)
          	80fa5a68-3843-4e25-b4c9-53a8c51ba9ff -- here (my laptop)
           	d642b276-5f9b-4a5c-a42e-ea29cd9540b5 -- usbdrive
    ok

Check the filesystem results

    marshal@home[~/annex]> ls -la ~/annex
    total 48
    drwxrwxr-x  3 marshal marshal  4096 Jul 19 14:59 .
    drwxr-xr-x 99 marshal marshal 24576 Jul 19 14:55 ..
    lrwxrwxrwx  1 marshal marshal   194 Jul 19 14:59 file1.m3u -> .git/annex/objects/jq/MW/SHA256E-s64061--c388e2c927cb4c88c11b1b5a8e166c4d8080dc528d53954dad1ee6d4933ce3c7.m3u/SHA256E-s64061--c388e2c927cb4c88c11b1b5a8e166c4d8080dc528d53954dad1ee6d4933ce3c7.m3u
    lrwxrwxrwx  1 marshal marshal   188 Jul 19 14:59 file2.sh -> .git/annex/objects/1J/2K/SHA256E-s318--b8729560cc759d2256903feaa4ba65994ccbac94f30515d52a39083ad52e1bad.sh/SHA256E-    s318--b8729560cc759d2256903feaa4ba65994ccbac94f30515d52a39083ad52e1bad.sh
    lrwxrwxrwx  1 marshal marshal   194 Jul 19 14:59 file3.pdf -> .git/annex/objects/77/Wj/SHA256E-s32239--9b051067d5ef9f0cb4a01750901bf6bee3f9348e5b10138c8ef416ac8d51e5df.pdf/SHA256E-s32239--9b051067d5ef9f0cb4a01750901bf6bee3f9348e5b10138c8ef416ac8d51e5df.pdf
    drwxrwxr-x  9 marshal marshal  4096 Jul 19 15:04 .git
    marshal@home[~/annex]> ls -la /media/marshal/Sony\ USB/annex/
    total 4
    drwx------ 1 marshal marshal  144 Jul 19 14:56 .
    drwx------ 1 marshal marshal 4096 Jul 19 14:56 ..
    drwx------ 1 marshal marshal  448 Jul 19 14:57 .git

Why is there nothing in the usbdrive repo? And worse, why is the whereis command saying there are 2 copies?

But if I sync from the usbdrive repo, the files are copied

    marshal@home[~/annex]> cd /media/marshal/Sony\ USB/annex/
    marshal@home[/media/marshal/Sony USB/annex]> git-annex sync laptop
    (merging synced/git-annex origin/git-annex into git-annex...)
    (Recording state in git...)
    commit  ok
    pull laptop
    remote: Counting objects: 23, done.
    remote: Compressing objects: 100% (8/8), done.
    remote: Total 11 (delta 3), reused 0 (delta 0)
    Unpacking objects: 100% (11/11), done.
    From /home/marshal/annex
     * [new branch]      git-annex  -> laptop/git-annex
     * [new branch]      master     -> laptop/master
     * [new branch]      synced/master -> laptop/synced/master

    Already up-to-date.
    ok
    (merging laptop/git-annex into git-annex...)
    (Recording state in git...)
    push laptop
    Counting objects: 56, done.
    Delta compression using up to 8 threads.
    Compressing objects: 100% (24/24), done.
    Writing objects: 100% (33/33), 2.34 KiB | 0 bytes/s, done.
    Total 33 (delta 15), reused 0 (delta 0)
    To /home/marshal/annex
     * [new branch]      git-annex -> synced/git-annex
    ok
    marshal@home[/media/marshal/Sony USB/annex]> ls -la /media/marshal/Sony\ USB/annex/
    total 6
    drwx------ 1 marshal marshal  456 Jul 19 15:07 .
    drwx------ 1 marshal marshal 4096 Jul 19 14:56 ..
    lrwxrwxrwx 1 marshal marshal  396 Jul 19 15:07 file1.m3u -> .git/annex/objects/jq/MW/SHA256E-s64061--c388e2c927cb4c88c11b1b5a8e166c4d8080dc528d53954dad1ee6d4933ce3c7.m3u/SHA256E-s64061--c388e2c927cb4c88c11b1b5a8e166c4d8080dc528d53954dad1ee6d4933ce3c7.m3u
    lrwxrwxrwx 1 marshal marshal  384 Jul 19 15:07 file2.sh -> .git/annex/objects/1J/2K/SHA256E-s318--b8729560cc759d2256903feaa4ba65994ccbac94f30515d52a39083ad52e1bad.sh/SHA256E-s318--b8729560cc759d2256903feaa4ba65994ccbac94f30515d52a39083ad52e1bad.sh
    lrwxrwxrwx 1 marshal marshal  396 Jul 19 15:07 file3.pdf -> .git/annex/objects/77/Wj/SHA256E-s32239--9b051067d5ef9f0cb4a01750901bf6bee3f9348e5b10138c8ef416ac8d51e5df.pdf/SHA256E-s32239--9b051067d5ef9f0cb4a01750901bf6bee3f9348e5b10138c8ef416ac8d51e5df.pdf
    drwx------ 1 marshal marshal  448 Jul 19 15:07 .git

Why did I have to sync from the usbdrive? I expected the repo setup and the --auto option to copy the files. What am I missing?

Any help much appreciated.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmcS2gG2R_AIiBBOsWuxGf1yEn_l797jjU"
 nickname="Mesar"
 subject="comment 1"
 date="2014-07-20T00:50:15Z"
 content="""

When you did:

    marshal@home[~/annex]> git-annex copy --auto --to usbdrive

this copies the file content (see .git/annex/objects/*), and updates the metadata in both repos.
The reason why you are not seeing the files on the usb is because the metadata has not been merged into your master branch.
This is verified by the output of:

    marshal@home[/media/marshal/Sony USB/annex]> git-annex sync laptop

To verify that you indeed do have the files and their content, don't do the above sync, but instead do:

    marshal@home[/media/marshal/Sony USB/annex]> git annex merge

the local metadata in the sync/git-annex branch will be merged in, and the local directory will be populated with the new symlinks to the content which you already have in the /media/marshal/Sony USB/annex/.git/annex/objects/*
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnwSZggfi3YE0EAuxAB9jT6pMcB73V8ae4"
 nickname="Marshal"
 subject="comment 2"
 date="2014-07-20T01:33:25Z"
 content="""
Thank you Mesar!

I think I'm beginning to understand. Indeed, the files were there on the usbdrive repo (in .git/annex/objects). No need to sync or merge. I was able to drop a file from the laptop repo and get it back from the usbdrive.

What is a bit unnerving, is that the symlinks are not present on the usbdrive, so it looks empty except for the .git directory.

Thanks so much for clearing this up. More to explore...
"""]]


code:
I have large central repositories of data. Therefore, on each client I want to save part data(to save space of disk). In command line I do

	[...]
	git-annex webapp
	git-annex drop [DeleteContentDirectory]
	[...]

After this command Assistant performs automatic synchronization getting content of files from this directory(DeleteContentDirectory), but I don't want. I want it's was only symlink of file in this directory.

How can I configure Assistant which files have to get content on the client?  It's possible?


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmWg4VvDTer9f49Y3z-R0AH16P4d1ygotA"
 nickname="Tobias"
 subject="comment 1"
 date="2013-08-26T07:44:22Z"
 content="""
You should set your repository to \"manual\" mode instead of \"client\" mode.

But then no data will be synced at all by the assistant, only metadata. You would have to do \"get/drop\" manually for commandline for all files.

Alternatively you could use the special \"archive\" folders that are supported by the assistant.

"""]]


code:
I have a setup as follows,

3 drives on a remote ssh server regular git annex repos.
One untrusted clone of this repository on my laptop.

On my laptop in webapp I have set num of copies to 2 (each file does have two copies on the server already). all repos plus laptop is set to manual mode. What happens is whenever I get a file to the laptop it would drop it from the disk on the server. Even though laptop is marked as untrusted this does not happen when using annex at the command line when I do a git annex get . --auto on the server it does not take the files on the untrusted repos into account which is what I want so I can drop files without moving them back to the server.

Is this the intended behavior or am I doing something wrong?


code:
Hey,

I write an article for the univerity about git-annex assistant and I'm a little bit confused about the function of the assistant.
Because an advantage of git-annex is that you can hold different files at different places in one repository, but this isn't anymore, or? The assistant syncs every file to every place of the repository. And this makes it just to another Dropbox (not direct the same) pentant or I'm not correct?

The transfer group is just for transfer files to other repositories, which are not connected with the network or?

I'm using ppa, ubuntu 12.04 lts, vers, 2013-12-13

JP Lührig


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="good question.."
 date="2013-12-19T17:04:53Z"
 content="""
The assistant necessarily simplifies some things git-annex does, so that it can automate them, and this includes keeping files synced to everywhere by default. There are ways to configure the assistant to only keep a subset of files on a computer, including [[archive_directories|assistant/archival_walkthrough]]. Underneath, there is the quite powerful [[preferred_content]] DSL, which can be used to configure lots more interesting behavior about what goes where. So far only some common scenarios have been made easily available in the webappp, but there's potential for lots more configurability here, if a good UI for it is built.

I think the assistant is probably better than dropbox at handling external removable media, used for backups or archiving. (But I have not evaluated, or actually even used dropbox myself. My goal is *not* to copy dropbox.) And it's better at encryption, and it allows using a variety of cloud services rather than only one provider.
"""]]


code:
I have 3 remotes for an annex - one on my laptop, one on a USB drive that's plugged in 50% of the time, and an rsync special remote on a friend's machine that I can access using SSH.

I have tried various things (annex copy, numcopies=3) to make it move data to the rsync remote, but it doesn't seem to want to work. I have tried git annex sync. I added the remote repo using the webapp, and it looked as if it was copying my files for a bit then stopped. I have the remote repo setup as a backup in the webapp.

The sizes of the directorys:
    ~ $ du -sh Documents
    126M	Documents
    ~ $ du -sh /Volumes/Backup/Documents
    227M	/Volumes/Backup/Documents

and the remote
    [c0g@womb Tom]$ du -sh MacDocuments/
    21M	MacDocuments/

I'm also curious as to why the usb drive repo is so much bigger than the thing I'm copying, but I'll worry about that later.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.7.238"
 subject="comment 1"
 date="2013-01-10T18:36:04Z"
 content="""
What does it say when you run:

  git annex copy --to $rsyncremote
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkjWgG_L90szs6o4E8cBusol57zkcDhg7c"
 nickname="Tom"
 subject="comment 2"
 date="2013-01-10T18:40:13Z"
 content="""
....
copy FrenchBasicCourserevised-Volume1-StudentText.pdf (checking Womb_MacDocuments...) ok
....

for each file in the directory.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.7.238"
 subject="comment 3"
 date="2013-01-10T18:44:00Z"
 content="""
Sounds like copy works then?

What version of git-annex do you have installed? A bug was fixed on December 11th, before that the assistant never stored things in repositories configured as backups.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkjWgG_L90szs6o4E8cBusol57zkcDhg7c"
 nickname="Tom"
 subject="comment 4"
 date="2013-01-10T18:48:16Z"
 content="""
Sorry for lack of extra information: The sizes of the remote folder is still at 21MB, well below even the size of the PDF I posted about (that's about 75MB). Copy claims to work, but I see no network use and the remote repo doesn't grow. I install using cabal on my max yesterday, git-annex version: 3.20130107

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.7.238"
 subject="comment 5"
 date="2013-01-10T18:58:49Z"
 content="""
Ok, so copy is checking if the file is present, and it seems to think it is, so doesn't try again to send it.

Here's what you need to do: Run `git annex copy FrenchBasicCourserevised-Volume1-StudentText.pdf  --to Womb_MacDocuments --debug`

It'll output something like this:

    copy new (checking rsync...) [2013-01-10 14:51:43 JEST] read: rsync [\"localhost:/tmp/r/f87/4d5/'SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'\"]

Now try running that command at the shell, and check its exit status. For the above, I ran:

    rsync \"localhost:/tmp/r/f87/4d5/'SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'\"; echo $?

It output:

    -rw-r--r--           0 2013/01/10 14:51:40 SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
    0

Compare with the same command but with the filename mangled.

    rsync: link_stat \"/tmp/r/f87/4d5/'SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/mangled'\" failed: No such file or directory (2)
    rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1536) [Receiver=3.0.9]
    27

It seems that rsync might be exiting 0 for content that's not there, in your case. This test will confirm or disprove that hypothesis and might provide some useful debug info.
"""]]