code:
**Premise:**

I've had this problem before 5.20140709-gc75193e and when the [ugly bug](http://git-annex.branchable.com/devblog/day_193-194__ugly_bug/) was discovered and fixed, I updated and thought it was gone.
Yet, I'm still experiencing a content removal bug, so it might be something different altogether.
I should also add that I don't know much about git log history and git-annex inner workings. I might need some help with finding the necessary information for debugging.

**Bug description:**

Two local computers, running Ubuntu-14.04 and Fedora-20 respectively, are paired through the assistant and functional with git annex 5.20140709. In direct mode. Most of the time, everything seems to work fine. Sometimes, *Computer A* will modify one file in its repository while *B* is offline. When *B* comes back online both repositories will lose their copy of the modified file (I'm not sure whether this also happens when they're both online when the change happens; it doesn't look like it anyway). The rest of the data in the repository is intact. This does not happen every time and I haven't been able to reproduce it consistently. Since 5.20140709 came out, it happened 2 or 3 times.

Looking at 'git log --stat' in *B*:

    commit d9eb9e94a39
    Author: COMPUTER B
    Date:   Thu Jul 17 18:17:16 2014 +0200
      shared.skg | 1 -
      1 file changed, 1 deletion(-)

    commit 6fa27f08492
    Author: COMPUTER A
    Date:   Thu Jul 17 18:17:14 2014 +0200
      shared.skg | 1 +
      1 file changed, 1 insertion(+)

    commit d25cc793739
    Author: COMPUTER A
    Date:   Thu Jul 17 18:17:12 2014 +0200
      shared.skg | 1 -
      1 file changed, 1 deletion(-)

A change is apparently deleting and rewriting the whole file (maybe that's normal because it's a binary (encrypted) file?). The third most recent deletion is probably the one that shouldn't happen.

Reverting "d9eb9e94a39" will only recover a placeholder, not really the original file. I can get the content back by going through *git annex unused* with *git log -S'key'*, then going to indirect mode, and then using *addunused*. This can only be done in the repository that changed the file in the first place (repo A), as the other repository only has the older version of the file in its unused list. This probably means that the change did not propagate before the file was deleted.

**daemon.log transcript**

[[!format sh """

[2014-07-17 18:13:17 CEST] main: starting assistant version 5.20140709-gc75193e
[2014-07-17 18:13:17 CEST] Cronner: You should enable consistency checking to protect your data.
[2014-07-17 18:13:55 CEST] TransferScanner: Syncing with asus.local_sincro
(scanning...) [2014-07-17 18:13:55 CEST] Watcher: Performing startup scan
p11-kit: couldn't load module: /usr/lib/x86_64-linux-gnu/pkcs11/p11-kit-trust.so: /usr/lib/x86_64-linux-gnu/pkcs11/p11-kit-trust.so: cannot open shared object file: No such file or directory
p11-kit: couldn't load module: /usr/lib/x86_64-linux-gnu/pkcs11/gnome-keyring-pkcs11.so: /usr/lib/x86_64-linux-gnu/pkcs11/gnome-keyring-pkcs11.so: cannot open shared object file: No such file or directory
(started...) p11-kit: couldn't load module: /usr/lib/x86_64-linux-gnu/pkcs11/p11-kit-trust.so: /usr/lib/x86_64-linux-gnu/pkcs11/p11-kit-trust.so: cannot open shared object file: No such file or directory
p11-kit: couldn't load module: /usr/lib/x86_64-linux-gnu/pkcs11/gnome-keyring-pkcs11.so: /usr/lib/x86_64-linux-gnu/pkcs11/gnome-keyring-pkcs11.so: cannot open shared object file: No such file or directory
gpg: Signature made gio 17 lug 2014 18:02:09 CEST using DSA key ID 89C809CB
gpg: /tmp/git-annex-gpg.tmp.0/trustdb.gpg: trustdb created
gpg: Good signature from "git-annex distribution signing key (for Joey Hess) <id@joeyh.name>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 4005 5C6A FD2D 526B 2961  E78F 5EE1 DBA7 89C8 09CB
[2014-07-17 18:13:58 CEST] Upgrader: An upgrade of git-annex is available.  (version 5.20140716)
Everything up-to-date
git-annex-shell: key is already present in annex
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [sender=3.1.0]

########## SYNCING STARTS HERE ###########

[2014-07-17 18:17:12 CEST] RemoteControl: Syncing with asus.local_sincro
From ssh://git-annex-asus.local-USER-A_22_sincro/~/sincro
   51815ea..d25cc79  annex/direct/master -> asus.local_sincro/annex/direct/master
   0391880..49efa5f  git-annex  -> asus.local_sincro/git-annex
   51815ea..d25cc79  master     -> asus.local_sincro/master
   51815ea..d25cc79  synced/master -> asus.local_sincro/synced/master
[2014-07-17 18:17:15 CEST] RemoteControl: Syncing with asus.local_sincro
Automatic merge went well; stopped before committing as requested
Removing shared.skg
From ssh://git-annex-asus.local-USER-A_22_sincro/~/sincro
   d25cc79..6fa27f0  annex/direct/master -> asus.local_sincro/annex/direct/master
   d25cc79..6fa27f0  master     -> asus.local_sincro/master
   d25cc79..6fa27f0  synced/master -> asus.local_sincro/synced/master
[2014-07-17 18:17:16 CEST] Committer: Committing changes to git
[2014-07-17 18:17:16 CEST] Pusher: Syncing with asus.local_sincro
To ssh://USER-A@git-annex-asus.local-USER-A_22_sincro/~/sincro/
   0391880..8d1c927  git-annex -> synced/git-annex
   6fa27f0..d9eb9e9  annex/direct/master -> synced/master
[2014-07-17 18:17:17 CEST] RemoteControl: Syncing with asus.local_sincro
From ssh://git-annex-asus.local-USER-A_22_sincro/~/sincro
   49efa5f..e328661  git-annex  -> asus.local_sincro/git-annex
[2014-07-17 18:17:20 CEST] RemoteControl: Syncing with asus.local_sincro
From ssh://git-annex-asus.local-USER-A_22_sincro/~/sincro
   6fa27f0..d9eb9e9  annex/direct/master -> asus.local_sincro/annex/direct/master
   e328661..32e44ce  git-annex  -> asus.local_sincro/git-annex

# End of transcript or log.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 1"
 date="2014-07-21T17:37:22Z"
 content="""
Well, this is definitely a different bug or issue than the \"ugly bug\". In particular, it only affects a single file. Also, based on the commits, this may not be occuring in a merge commit at all (although I am not sure; I am not familiar with the log syntax shown).

AFAICS, the problem occurs on machine B. Which machine is the transcript from?

The \"Removing shared.skg\" is a good clue. This seems to be printed by `git`, in merge-recursive.c's `process_entry`. What is puzzling to me is that it's printed after the \"Automatic merge went well; stopped before committing as requested\", which  AFAICS is printed out by git last thing. Is this \"Removing\" message then printed out by another git command?

Enabling debug logging would probably help a lot, to narrow that down the next time this occurs.
"""]]

repo_name: kakafarm/git

license: unknown


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmMLeU-zCzx2mc5pL2XT8a1UNkQwHAHjg8"
 nickname="daniele"
 subject="comment 2"
 date="2014-07-22T01:47:29Z"
 content="""
> I am not familiar with the log syntax shown

You mean the \"git log --stat\" part? Which git command would yield the most helpful syntax in this case?

> AFAICS, the problem occurs on machine B. Which machine is the transcript from?

Sorry I forgot to mention it: yes, it's from machine B.

> Is this \"Removing\" message then printed out by another git command?

Sorry I have no clue here. I didn't issue any git command from the terminal (nor did the user on computer A) if that was part of the question. It was all done in automatic.

> Enabling debug logging would probably help a lot, to narrow that down the next time this occurs.

Will do. I'll set 'annex.debug' to true in .git/config. Sadly, computer A is (a laptop) on vacation at the moment (well outside the local network), so I'll have to wait a couple of weeks to get back to debugging this. I'll have the logs with debug enabled when it happens again.

Thanks again for your support and for developing git-annex.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmMLeU-zCzx2mc5pL2XT8a1UNkQwHAHjg8"
 nickname="daniele"
 subject="comment 3"
 date="2014-07-22T02:10:14Z"
 content="""
I was unsure as to what git-log command would best describe the problematic commits but in the meantime I did a:

     git log --graph --decorate=full --full-diff

These are the only three commits of that afternoon (the surrounding history is from completely different hours and very likely unrelated, so it wasn't posted)

    * commit d9eb9e94a3973598a847a5bdab1b65e459c1588a
    | Author: COMPUTER B
    | Date:   Thu Jul 17 18:17:16 2014 +0200
    |
    * commit 6fa27f0849227c490ac4d4d62ca86e4befe5121e
    | Author: COMPUTER A
    | Date:   Thu Jul 17 18:17:14 2014 +0200
    |
    * commit d25cc793739573057e475c92c8d37ce4ecc7bc9b
    | Author: COMPUTER A
    | Date:   Thu Jul 17 18:17:12 2014 +0200

It's a straight line (fast forward?), I don't see any merging either. Is this normal? Shouldn't a change in Repo A bring a merge in Repo B (where everything stayed the same) when things are synchronized? I don't fully understand how annex syncs happen so don't mind this question if it's all normal.
"""]]


code:
### Please describe the problem.
When "git annex sync --content" is used only objects currently in the working tree are synced. It doesn't honor full archives, which should get all objects.
So only objects similar to "git annex find --want-get" are synced and not every available object "git annex get --all"

### What steps will reproduce the problem?
# mein repo:
git annex add file
git annex sync
git annex rm file
git annex sync

# other repo:
git annex wanted here "not copies=backup:3"
git annex find --want-get # will be empty
git annex sync --content # nothing is synced even so all files should be backed up
git annex get --all # will sync the object from file

### What version of git-annex are you using? On what operating system?
git-annex version: 5.20140210-gd99db49
Linux (Ubuntu 13.10)

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

# End of transcript or log.
"""]]

[[!tag confirmed]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.172"
 subject="comment 1"
 date="2014-02-20T19:34:29Z"
 content="""
Yep, sync --content only looks at the work tree.

I suppose that the assistant does a better job in this situation, since if it cannot access an archive repo, it will remember that it tried to send a file to it, and retry that transfer later -- even if in the meantime the file has gotten deleted out of the work tree (and still has content present, due to using indirect mode). I'm actually not 100% sure .. the assistant may give up on transferring a file if it's gotten removed from the work tree. It's worth considering this because I basically want sync --content to do the same syncing that the assistant does.

Anyway, sync --content could certianly look at all keys present in the annex. This would require a separate pass, and it might then try to upload a key twice, once from work tree, and once from annex, and fail twice.

Maybe it would be better to have this as a separate --content --all. It might also make sense to keep sync --content only looking at the work tree by default to support cases where there are multiple branches and you only want to sync one.

I think this needs more thought.
"""]]


code:
[[!comment format=mdwn
 username="stp"
 ip="84.56.21.11"
 subject="Some ideas for content sync"
 date="2014-02-20T21:18:30Z"
 content="""
I agree that it is a tricky situation. The main issue I have is that archive preferred content expressions indicates that all versions are transferred, but when you (worst case) migrate full archives to other disks and just use git annex sync --content, you would loose all objects.

I like the idea of --content --all as this would be consistent with other commands such as fsck for example.
Still it seems with preferred content expressions only applying to working tree data that it would still miss \"copies=backup:2\" for example and only use working tree files. Which is misleading and could lead to dataloss in my opinion.

So the best would probably be to let preferred content for number of copies at least work on all objects, either per default or when \"git annex sync --content --all\" is used. You wouldn't run that on usual repos, but definitely on backup, archive or similar repos.

"""]]


code:
[[!comment format=mdwn
 username="stp"
 ip="84.56.21.11"
 subject="Ídea"
 date="2014-02-23T14:25:22Z"
 content="""
I thought about the implementation need for git annex sync --content --all. If preferred content expressions would work it would be needed. Everything else. could be done via a split usage.
Run \"git annex sync --content\" to satisfy the preferred content expressions on the working tree and the numcopies on the working tree and then loop through all backup/archive repositories with \"git annex get --auto\" this should at least prevent archives from getting objects numcopies is already satisfying and sync the objects not yet satisfied right?
"""]]


code:
### Please describe the problem.

I plugged in a usb drive, did git annex sync and it git fast-forward and deleted about 600 files that I had added on my laptop.

### What steps will reproduce the problem?

Obviously, I'm not sure really because I don't plug this usb drive every day to sync so I don't remember what I did last time. But I suppose I just finished with git annex sync and unplugged it. When the accident occured:

 1. The usb drive was in direct mode whereas the laptop was in indirect mode at the time.
 2. I git annex sync the usb drive
 3. I git annex sync the laptop

Anyway, the big mistake I did was syncing the laptop as well, naively thinking it would correct the usb drive; but instead it also deleted the files on the laptop. I had a back up of most of it so it was okay.

---

So now I git reset --hard to a commit before I synced, so I still have my files. But how can I fix this situation?

### What version of git-annex are you using? On what operating system?

Arch Linux

git-annex version: 5.20140128
build flags: S3 DBus TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web glacier hook external
local repository version: 5
supported repository version: 5
upgrade supported from repository versions: 0 1 2 4

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

# End of transcript or log.
"""]]

> [[closing|done]], not a bug based on the limited description. --[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawno-jcsScu4CK6k2QLZqxMros1PQHf1NQY"
 nickname="Hugo"
 subject="Sync messed up"
 date="2014-03-09T12:16:32Z"
 content="""
So, I have now reseted to a previous commit all the branches: git-annex, master, synced/git-annex and synced/master in other usb drives. i can git pull and git push, etc. But every time I try a git annex sync, it's deleting files again.

For instance:

```````````````
(merging laptop/git-annex laptop/synced/git-annex into git-annex...)
(Recording state in git...)
commit  ok
pull wdrouge
Depuis /run/media/hrd/WD-rouge/annex/hrd
 * [nouvelle branche] git-annex  -> wdrouge/git-annex
   e5894a1..f5af709  master     -> wdrouge/master
 * [nouvelle branche] synced/git-annex -> wdrouge/synced/git-annex
 * [nouvelle branche] synced/master -> wdrouge/synced/master
ok
pull origin
Depuis /home/hrd
 + 93d883b...f5af709 git-annex  -> origin/git-annex  (mise à jour forcée)
   e5894a1..f5af709  master     -> origin/master
 + c8c2481...f5af709 synced/git-annex -> origin/synced/git-annex  (mise à jour forcée)
   1d2a028..ac708e3  synced/master -> origin/synced/master
 * [nouvelle étiquette] should-be-fine-here -> should-be-fine-here

Mise à jour f5af709..ac708e3
Fast-forward

→ a bunch of files
 621 files changed, 22 insertions(+), 599 deletions(-)
 delete mode 120000 → the bunch of files……………
 …
 delete mode 120000 org/gtd.org_archive
ok
pull laptop
Depuis /home/hrd
 + 93d883b...f5af709 git-annex  -> laptop/git-annex  (mise à jour forcée)
 + c8c2481...f5af709 synced/git-annex -> laptop/synced/git-annex  (mise à jour forcée)
   1d2a028..ac708e3  synced/master -> laptop/synced/master

Already up-to-date.
ok
push wdrouge
Counting objects: 6609, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3057/3057), done.
Writing objects: 100% (3331/3331), 511.27 KiB | 0 bytes/s, done.
Total 3331 (delta 2091), reused 0 (delta 0)
To /run/media/hrd/WD-rouge/annex/hrd
   f5af709..16f17bf  git-annex -> synced/git-annex
   f5af709..ac708e3  master -> synced/master
ok
push origin
Counting objects: 6569, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3056/3056), done.
Writing objects: 100% (3330/3330), 511.05 KiB | 0 bytes/s, done.
Total 3330 (delta 2091), reused 0 (delta 0)
To /home/hrd/
   f5af709..16f17bf  git-annex -> synced/git-annex
ok
push laptop
Everything up-to-date
ok
git annex sync  14.33s user 1.87s system 74% cpu 21.696 total
``````````````````

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 2"
 date="2014-03-10T17:11:41Z"
 content="""
Your laptop is in indirect mode, so we know that the only way files can be deleted by a merge is if a commit was made to git that deletes the files.

My conclusion is that some repository, perhaps the usb drive, made a commit that deleted those files. You should be able to find this commit with `git log --stat`, and can just `git revert` it if you want to.

So far, I don't see evidence of a bug. For all I know, you actually did delete the files on the usb drive, and that change got committed..
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawno-jcsScu4CK6k2QLZqxMros1PQHf1NQY"
 nickname="Hugo"
 subject="comment 3"
 date="2014-03-13T14:36:20Z"
 content="""
> My conclusion is that some repository, perhaps the usb drive, made a commit that deleted those files. You should be able to find this commit with git log --stat, and can just git revert it if you want to.

It would be surprising if I did that.

Anyway, I was not able to find which commit deleted the ~600 files. I just decided to re-start completely with git annex :-/

The good thing is that I did not lose any file, so in that regard git annex is great ;^)

However, one thing that is quite confusing to me is the way git annex [sync] works.  Am I supposed to run git annex sync in every repository? Because if I just run it once in 1 repo, then I usually don't get all the syncing done. Maybe I just don't understand something.

Thanks for replying,

[sync]: http://git-annex.branchable.com/sync/
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.154"
 subject="comment 4"
 date="2014-03-13T15:54:28Z"
 content="""
Did you run `git log --stat` and look for a commit that deleted a lot of files?
"""]]


code:
### Please describe the problem.

In a direct mode repo (crippled/uncrippled filesystem does not matter), when a symlink is marked using `git update-index --skip-worktree <FILE>` and removed, git annex sync still `git rm`s the symlink. This does not happen in indirect mode (git annex sync leaves the symlink in git intact).

### What steps will reproduce the problem?

[[!format  sh """
mkdir test-repo; cd test-repo
git init
git annex init
echo file1 >file1
git annex add
git commit -m"update"
cd ..
git clone test-repo test-repo2; cd test-repo2
git annex init
git annex direct
git update-index --skip-worktree file1
rm file1
git annex sync
"""]]

Output of `git annex sync` indicates file has been removed from git. Repeating these steps without the `git annex direct` above to set the second repo to direct mode will succeed in retaining the symlink in git.

### What version of git-annex are you using? On what operating system?

4.20130521 using git-annex-standalone AUR build (uses Linux executable tarball) on Arch Linux

### Please provide any additional information below.

I'd like to use the skip-worktree scheme in order to be able to rm the symlink files (from the filesystem, not git), specifically for my Android devices. Syncing my music annex creates .mp3 symlinks that aren't actually MP3s, which gives the stock apps some fits. This would only be for clearing out symlinks; I fully understand that trying to doing this for downloaded content in a direct repo would be a Class A no-no. :-)

I did a little digging in the code, and it looks like the source of this is the stageDirect step done specifically by `git annex sync` in direct repos (which makes sense, since indirect repos work). It does `git ls-files --others --exclude-standard --stage`. This list includes files marked skip-worktree, which means skip-worktree files would be treated like normal, and deleted because it's no longer there. There is an additional `-t` argument that could be added to ls-files that would provide the tag field to indicate if a file was marked skip-worktree, and they could be filtered out of processing.

I wonder if this would have side effects, or if there are other places in the code where skip-worktree files would need to be handled, though. I'm particularly motivated to solve this, so let me know if it doesn't look like it would get looked at right away, and I'll have an excuse to get a Haskell dev environment setup again and shake the rust off.

[[!tag confirmed]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-05-29T16:45:03Z"
 content="""
I'm a little worried about -t being \"semi-deprecated\". I don't know if it would be possible to use either of the other commands the man page suggests to get the info needed by `stageDirect`. (Particularly the Sha of their staged contents.)

All of git-annex's access to the repository tree goes via Git.LsFiles and AFAICS, all the other git commands used there do respect skip-worktree.

I encourage you to work on this, since you're motivated.
"""]]


code:
[[!comment format=mdwn
 username="EvanDeaubl"
 ip="12.130.123.174"
 subject="comment 2"
 date="2013-05-31T13:38:43Z"
 content="""
I share the concerns about the semi-deprecated status, although it's probably safer than they make it sound in the man page. The git test suite for sparse checkouts uses the same command to find files with skip-worktree set, and there are other tools that use it as well (magit being one example). The git maintainers couldn't remove it, or even fully deprecate it for eventual removal. It's been semi-deprecated for almost 3 years.

I have a patch that fixes `git annex sync` and `git annex get` (it retrieved files marked skip-worktree) and it passes `git annex test`, but I'm going to more rigorously test it out on my particular use case before calling it good. When it is ready, I didn't see instructions on how you would like patches submitted anywhere in the wiki. How would you like to receive it?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 3"
 date="2013-05-31T18:22:12Z"
 content="""
I accept patches by email, or pull requests.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.102"
 subject="ping?"
 date="2014-03-19T20:54:57Z"
 content="""
You had a patch, but never shared it. I'm curious to see it..
"""]]


code:
[[!comment format=mdwn
 username="EvanDeaubl"
 ip="24.251.129.149"
 subject="comment 5"
 date="2014-04-09T03:28:24Z"
 content="""
I'm afraid I abandoned this patch. It worked, but was still fidgety for being able to ignore parts of the tree. I found another way to do what I wanted by loading an indirect repo into /data and taking advantage of a surprise side effect in how the /sdcard filesystem translated the symlinks from the ext4 filesystem.

I can probably scare it up from my archives, but it hasn't been kept up to date. The good news is (as I recall) the patch was pretty small.

"""]]


code:
Running a command like

git annex unlock myfile

is not atomic, that is if the execution is aborted you may end up with an incomplete version of myfile in the directory. If you don't notice this you may lock it again and then propagate this bad version of the file to your other repositories. A simple workaround is to simply name it something else while unlocking and then rename it to the correct filename once it's completely copied. I don't know Haskel yet so I can not fix this issue otherwise I would sure try. A part from this, I love git annex.

> [[fixed|done]] --[[Joey]]


code:
[[!meta title="`git annex unused` fails on empty repository"]]

The ``git annex unused`` command fails on a git-annex repository, if there are no objects yet:

    $ git annex unused
    unused  (checking for unused data...)
    git-annex: /tmp/annextest/other_annex/.git/annex/objects: getDirectoryContents: does not exist (No such file or directory)
    git-annex: 1 failed
    $

This can give a user (especially one that wants to try out simple commands with his newly created repo) the impression that something is wrong, while it is not. I'd expect the program either to show the same message ``git annex unused`` shows when everything is ok (since it is, or should be).

This can be a bug in the ``unused`` subcommand (that fails to accept the absence of an objects directory) or in the ``init`` subcommand (that fails to create it).

> [[fixed|done]] --[[Joey]]


code:
"git annex upgrade" has lost track of some of my files.  Most of them have "&" characters.  The others contain "%" characters (I haven't tried the testcase below with "%" however).

Testcase:

    # (With git annex v2)
    mkdir ~/testannex1
    cd ~/testannex1
    git init
    git annex init "testannex1"
    touch '02 - Afternoons & Coffeespoons.mp3'
    touch 'no ampersand.mp3'
    git annex add '02 - Afternoons & Coffeespoons.mp3'
    git annex add 'no ampersand.mp3'
    git commit -m added
    git annex whereis '02 - Afternoons & Coffeespoons.mp3'
    git annex whereis 'no ampersand.mp3'
    # (Upgrade git-annex binary to v3 and then...)
    git annex upgrade
    git annex whereis '02 - Afternoons & Coffeespoons.mp3'
    git annex whereis 'no ampersand.mp3'

This produces:

    12:38:40 ~/testannex1 (master)$ git annex whereis '02 - Afternoons & Coffeespoons.mp3'
    whereis 02 - Afternoons & Coffeespoons.mp3 (0 copies)
    failed
    git-annex: 1 failed
    12:38:40 ~/testannex1 (master)$ git annex whereis 'no ampersand.mp3'
    whereis no ampersand.mp3 (1 copy)
      	a7b680fc-a8d0-11e0-b0fe-4f94e86d1fb7  -- testannex1 <-- here
    ok

[[!tag done]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2011-07-07T21:04:23Z"
 content="""
What an evil little bug. In retrospect, this probably bit my own test upgrades, but I ran `git annex fsck` everywhere and so avoided the location log breakage.

I've fixed the bug, which also involved files with other punctuation in their names [&:%] when using the WORM backend.

The only way I have to recover repos that have already been upgraded is to run `git annex fsck --fast` in each clone of such a repo, which will let it rebuild the location log information. I think that is the best way to recover; ie I can't think of a way to recover that doesn't need to do everything fsck does anyway.
"""]]


code:
### Please describe the problem.

webapp needs to be killed and restarted to finish setting up a new repository

### What steps will reproduce the problem?

I run on a remote linux server

git annex webapp --listen=10.222.0.1:4000

I get a url printed

click the url, it opens in my browser
click make a repository, and it doesn't finish loading the web page
if I ctrl-c git on the remote server and start it up again, click
the url again, i can continue to set up the new repository

### What version of git-annex are you using? On what operating system?

git-annex version: 4.20130709
build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP

debian wheezy with git-annex pinned from sid

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

[2013-07-19 08:09:54 EST] main: starting assistant version 4.20130709
WebApp crashed: unable to bind to local socket
[2013-07-19 08:09:54 EST] WebApp: warning WebApp crashed: unable to bind to local socket

  dbus failed; falling back to mtab polling (ClientError {clientErrorMessage = "runClient: unable to determine DBUS address", clientErrorFatal = True})

  No known network monitor available through dbus; falling back to polling
(scanning...) [2013-07-19 08:09:54 EST] Watcher: Performing startup scan
(started...) Merge made by the 'recursive' strategy.
 ...book.azw |    1 +
 1 file changed, 1 insertion(+)
 create mode 120000 Books/book.azw
[2013-07-19 08:13:03 EST] Committer: Committing changes to git

# End of transcript or log.
"""]]

> Duplicate; [[closed|done]]. --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.140"
 subject="comment 1"
 date="2013-07-20T19:57:05Z"
 content="""
This is a known problem with using --listen with a forced port number. See [[Hangs_on_creating_repository_when_using_--listen]]
"""]]


code:
### Please describe the problem.

When I `git mv` a file that I just `git annex add`-ed without having `git commit`-ed it first, then the symlink will break.

### What steps will reproduce the problem?

    $ mkdir foo
    $ echo hello > foo/bar
    $ git annex add foo/bar
    $ git mv foo/bar .

### What version of git-annex are you using? On what operating system?

5.20140529 on Debian testing

> This is fundamentally something git-annex cannot deal with,
> because there is no way to hook into git to fix the symlink when
> `git mv` moves the file.
>
> Instead, git-annex has several "good enough" fixes for the problem:
>
> * As soon as you `git commit`, the pre-commit hook will run `git annex
>   fix` and this will fix the symlink before it gets committed.
> * You can run `git annex fix` yourself after `git mv`.
> * Even `git annex add $file` will fix the symlink if the file is already
>   annexed.
>
> So, [[done]] --[[Joey]]


code:
It's unfortunate that git-annex sorta defeats git's rename detection.

When an annexed file is moved to a different directory (specifically, a
directory that is shallower or deeper than the old directory),
the symlink often has to change. And so git log cannot --follow back
through the rename history, since all it has to go on is that symlink,
which it effectively sees as a one line file containing the symlink target.

One way to fix this might be to do the `git annex fix` *after* the rename
is committed. This would mean that a commit would result in new staged
changes for another commit, which is perhaps startling behavior.

The other way to fix it is to stop using symlinks, see [[todo/smudge]].
[[!tag confirmed]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnpdM9F8VbtQ_H5PaPMpGSxPe_d5L1eJ6w"
 nickname="Rafael"
 subject="comment 10"
 date="2012-05-15T07:36:25Z"
 content="""
Won't git itself be fixed on this issue? It was on my plans to look into that, however I don't know how difficult it will be.
"""]]


code:
[[!comment format=mdwn
 username="http://christian.amsuess.com/chrysn"
 nickname="chrysn"
 subject="use mini-branches"
 date="2011-03-09T23:47:48Z"
 content="""
if you go for the two-commits version, small intermediate branches (or git-commit-tree) could be used to create a tree like this:

    *   commit 106eef2
    |\  Merge: 436e46f 9395665
    | |
    | |     the main commit
    | |
    | * commit 9395665
    |/
    |       intermediate move
    |
    * commit 436e46f
    |
    |     ...

while the first commit (436e46f) has a \"`/subdir/foo → ../.git-annex/where_foo_is`\", the intermediate (9395665) has \"`/subdir/deeper/foo → ../.git-annex/where_foo_is`\", and the inal commit (106eef2) has \"`/subdir/deeper/foo → ../../.git-annex/where_foo_is`\".

`--follow` uses the intermediate commit to find the history, but the intermediate commit would neither show up in `git log --first-parent` nor affect `git diff HEAD^..` & co. (there could still be confusion over `git show`, though).
"""]]


code:
[[!comment format=mdwn
 username="praet"
 ip="81.240.159.215"
 subject="Use variable symlinks, relative to the repo's root ?"
 date="2011-03-10T16:50:28Z"
 content="""
It all boils down to the fact that the path to a relative symlink's target is determined relative to the symlink itself.

Now, if we define the symlink's target relative to the git repo's root (eg. using the $GIT_DIR environment variable, which can be a relative or absolute path itself), this unfortunately results in an absolute symlink, which would -for obvious reasons- only be usable locally:

    user@host:~$ mkdir -p tmp/{.git/annex,somefolder}
    user@host:~$ export GIT_DIR=~/tmp
    user@host:~$ touch $GIT_DIR/.git/annex/realfile
    user@host:~$ ln -s $GIT_DIR/.git/annex/realfile $GIT_DIR/somefolder/file
    user@host:~$ ls -al $GIT_DIR/somefolder/
    total 12
    drwxr-x--- 2 user group 4096 2011-03-10 16:54 .
    drwxr-x--- 4 user group 4096 2011-03-10 16:53 ..
    lrwxrwxrwx 1 user group   33 2011-03-10 16:54 file -> /home/user/tmp/.git/annex/realfile
    user@host:~$

So, what we need is the ability to record the actual variable name (instead of it's value) in our symlinks.

It *is* possible, using [variable/variant symlinks](http://en.wikipedia.org/wiki/Symbolic_link#Variable_symbolic_links), yet I'm unsure as to whether or not this is available on Linux systems, and even if it is, it would introduce compatibility issues in multi-OS environments.

Thoughts on this?
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 3"
 date="2011-03-16T03:03:19Z"
 content="""
Interesting, I had not heard of variable symlinks before. AFAIK linux does not have them.
"""]]


code:
[[!comment format=mdwn
 username="praet"
 ip="81.240.27.89"
 subject="Brainfart"
 date="2011-03-20T20:11:27Z"
 content="""
Haven't given these any serious thought (which will become apparent in a moment) but hoping they will give birth to some less retarded ideas:

---

### Bait'n'switch

- pre-commit: Replace all staged symlinks (when pointing to annexed files) with plaintext files containing the key of their respective annexed content, re-stage, and add their paths (relative to repo root) to .gitignore.
- post-commit: Replace the plaintext files with (git annex fix'ed) symlinks.

In doing so, the blobs to be committed can remain unaltered, irrespective of their related files' depth in the directory hierarchy.

To prevent git from reporting ALL annexed files as unstaged changes after running post-commit hook, their paths would need to be added to .gitignore.

This wouldn't cause any issues when adding files, very little when modifying files (would need some alterations to \"git annex unlock\"), BUT would make git totally oblivious to removals...

---

### Manifest-based (re)population
- Keep a manifest of all annexed files (key + relative path)
- DON'T track the symlinks (.gitignore)
- Populate/update the directory structure using a post-commit hook.

... thus circumventing the issue entirely, yet diffstats (et al.) would be rather uninformative.

---

***Wide open to suggestions, criticism, mocking laughter and finger-pointing :)***
"""]]


code:
[[!comment format=mdwn
 username="praet"
 ip="81.242.56.203"
 subject="comment 5"
 date="2011-03-21T19:58:34Z"
 content="""
In the meantime, would it be acceptable to split the pre-commit hook
into two discrete parts?

This would allow to (if preferred) defer \"git annex fix\" until
post-commit while still keeping the safety net for unlocked files.
"""]]


code:
[[!comment format=mdwn
 username="http://adamspiers.myopenid.com/"
 nickname="Adam"
 subject="extra level of indirection"
 date="2011-12-19T12:45:18Z"
 content="""
Surely this could be handled with an extra layer of indirection?

git-annex would ensure that every directory containing annexed data contains a new symlink `.git-annex` which points to `$git_root/.git/annex`.  Then every symlink to an annexed object uses a relative symlink via this: `.git_annex/objects/xx/yy/ZZZZZZZZZZ`.  Even though this symlink is relative, moving it to a different directory would not break anything: if the move destination directory already contained other annexed data, it would also already contain `.git-annex` so git-annex wouldn't need to do anything.  And if it didn't, git-annex would simply create a new `.git-annex` symlink there.

These `.git-annex` symlinks could either be added to `.gitignore`, or manually/automatically checked in to the current branch - I'm not sure which would be best.  There's also the option of using multiple levels of indirection:

    foo/bar/baz/.git-annex -> ../.git-annex
    foo/bar/.git-annex -> ../.git-annex
    foo/.git-annex -> ../.git-annex
    .git-annex -> .git/annex

I'm not sure whether this would bring any advantages.  It might bring a performance hit due to the kernel having to traverse more symlinks, but without benchmarking it's difficult to say how much.  I'd expect it only to be an issue with a large number of deep directory trees.
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 7"
 date="2011-12-19T18:22:25Z"
 content="""
That seems an excellent idea, also eliminating the need for git annex fix after moving.

However, I think CVS and svn have taught us the pain associated with a version control system putting something in every subdirectory. Would this pain be worth avoiding the minor pain of needing git annex fix and sometimes being unable to follow renames?
"""]]


code:
[[!comment format=mdwn
 username="http://adamspiers.myopenid.com/"
 nickname="Adam"
 subject="comment 8"
 date="2011-12-20T12:00:11Z"
 content="""
Personally I'd rather have working rename detection but I agree it's not 100% ideal to be littering multiple directories like this, so perhaps you could make it optional, e.g. based on a git config setting?

Here are a few more considerations, some in defence of the approach, some against it:

* `.git-annex` is hidden; `CVS/` is not.
* Unlike `CVS/` and `.svn/`, it's only a symlink, not a directory containing other files.
* It doesn't contain any data specific to that directory and could easily be regenerated if deleted accidentally or otherwise.
* If a whole directory containing `.git-annex` was moved within the repository:
    * git-annex would need to fix up these symlinks if and only if it's moved to a different depth within the tree.
    * However, if the multi-level indirection approach is used, `.git-annex` in any subdirectory is *always* a symlink to `../.git-annex` so instead you would need to check that all of the new ancestors contain this symlink too, and optionally remove any no longer needed symlinks.
    * In either case, git-annex already goes to the trouble of fixing symlinks, and if anything, I *think* this approach would reduce the number of symlinks which need checking (right?)
* find `$git_root/foo -follow`, `diff -r` etc. would traverse into `$git_root/.git/annex`

This last point is the only downside to this approach I can think of which gives me any noticeable cause for concern.  However, people are already use to working around this from CVS and svn days, e.g. `diff -r -x .svn` so I don't think it's anywhere near bad enough to rule it out.
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 9"
 date="2011-12-20T14:56:12Z"
 content="""
Git can follow the rename fine if the file is committed before `git annex fix` (you can git commit -n to see this), so
making git-annex pre-commit generate a fixup commit before the staged commit would be one way. Or the other two ways I originally mentioned when writing down this minor issue. I like all those approaches better than .git-annex clutter.
"""]]


code:
### Please describe the problem.

When git-annex assistant on Android setups up the DCIM repo a .gitignore file is dropped into place which ignores .thumbnails/* . Unfortunately this doesn't match .thumbnails/.thumbdata* - I have a 700MB file which matches this pattern.

I suspect that making the pattern in .gitignore just .thumbnails should resolve this.

### What steps will reproduce the problem?

Create a repo

### What version of git-annex are you using? On what operating system?

Nightly build for Android from yesterday (2013-11-16)

### Please provide any additional information below.

> [[fixed|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.64"
 subject="comment 1"
 date="2013-11-16T22:45:52Z"
 content="""
\".thumbnails/*\" in .gitignore makes git ignore dotfiles in the directory in my testing, and according to git's own documentation.

However, it is the case that subdir/.thumbnails/* are not ignored by \".thumbnails/*\", but \".thumbnails\" will indeed ignore .thumbnails directories at any point in the tree. So, I've made the change you suggested.

(I also tested on android, with 1.8.4.1.559 and the assistant was able to honor the .gitignore, and did not add any files from `.thumbnails/`.)
"""]]


code:
glacier-cli currently relies on a local cache of
inventory information, and so other git-annexes using the same glacier
repository are not able to access stuff in it, unless and until
`glacier vault sync` is run.

An example of this causing trouble is with the assistant. When a file is
moved into archive/, the assistant that sends it to glacier is able to
trust that it's in glacier and remove the local copy. But other assistants
that also have a copy cannot trust that, and so don't remove their copies.

I've discussed with glacier-cli's author making git-annex store enough info
in its branch to be able to bootstrap glacier-cli to know about a file.
This seems doable and he had a design; waiting on movement
on the glacier-cli side.

[[!tag confirmed]]


code:
### Please describe the problem.
Git-Annex crashes when configuring jabber account with foo.bar@googlemail.com

### What steps will reproduce the problem?
Configure the Jabber Account with foo.bar@googlemail.com instead of foo.bar@gmail.com. The domain googlemail was used for a long time in germany because of a license issue.

### What version of git-annex are you using? On what operating system?
Mac OS X - 10.8.3 Mountain Lion

Version: 4.20130709-g18e5f43

Build flags: Assistant Webapp Pairing Testsuite S3 WebDAV FsEvents XMPP DNS

[[!meta title="xmpp fails to work with googlemail domain"]]
[[!tag design/assistant]]