code:
### Please describe the problem.

I had set up git-annex on a mac; I had created an initial repository at ~/annex; I had created a second repository on an external drive, at /Volumes/Biblio/annex; I had paired with three other machines on the same network, (two linux, one other mac) and set up a remote server as a backup-type repository. All seemed well. It had finally finished syncing everything to the remote server (my upload speeds are slow).

I closed the firefox window showing the dashboard. I wanted to reopen it, so I ran the git-annex.app again, presuming on a running instance that that just opens the browser back at the webapp. Firefox window opened, but the only repository was the second one I'd made on the external drive.

I restarted, as best as I could work out: git-annex assistant --stop, then because that left behind a process, killall git-annex. Then restarted the app.

Firefox opened on the webapp. I had two repositories: The one on the external drive (now the "Here" repo) and the one on ~/annex but only as if it was paired from a different machine.

ie: I see only "celestia.local (rachel@celestia.local~/annex)". This machine *is* celestia.local.

That's it. Startup scan took a couple of minutes but didn't add anything. Then it decided to sync to celestia.local, which it took a little time over but didn't apparently do anything.

If I drop files into ~/annex they are not synced anywhere. ~/annex still has a .git directory, populated with git files, it looks intact. It's just not being seen.

Is it possible because the user is prompted to create their initial repo at ~/Desktop/annex it will by default only look there, then start looking in external drives for it? So the fact I didn't want it on my desktop, but put it directly in home, meant it got lost on restart?

git-annex vicfg in ~/annex shows me this:

[[!format sh """
# git-annex configuration
#
# Changes saved to this file will be recorded in the git-annex branch.
#
# Lines in this file have the format:
#   setting uuid = value

# Repository trust configuration
# (Valid trust levels: trusted semitrusted untrusted dead)
# (for web)
#trust 00000000-0000-0000-0000-000000000001 = semitrusted
# (for rachel@octavia:~/annex)
#trust 161dec38-e8be-43b8-86c5-555d35ce3416 = semitrusted
# (for rachel@celestia.local:~/annex)
#trust 179fcddf-e247-4577-804b-267feed8abb1 = semitrusted
# (for 192.168.1.103_annex (rachel@rainbow.local:~/annex))
#trust 256d5762-150d-4d5d-9340-517de298c874 = semitrusted
# (for twilight.local_annex (rachel@twilight:~/annex))
#trust aeef7490-ce27-4255-b800-1947706c4a06 = semitrusted
# (for rachel@octavia:~/annex)
#trust c469fbce-f3b4-4e27-a54f-0b747797a7d5 = semitrusted
# (for annex (Biblio's Copy))
#trust c9e307e2-1189-47ed-8ad4-03b5c1b64e36 = semitrusted
# (for luna.strangenoises.org_annex)
#trust f36dbdf8-1bba-11e3-9dbe-f33cfb0e2bed = semitrusted
# (for octavia.local_annex (rachel@octavia:~/annex))
#trust f748a5ed-d870-48fb-b3ec-811488eb2faa = semitrusted
# (for rachel@twilight:~/annex)
#trust fcaba03e-1ba5-11e3-90f1-57fe1467e006 = semitrusted

# Repository groups
# (Standard groups: client transfer backup incrementalbackup smallarchive archive source manual public unwanted)
# (Separate group names with spaces)
# (for rachel@octavia:~/annex)
group 161dec38-e8be-43b8-86c5-555d35ce3416 = client
# (for rachel@celestia.local:~/annex)
group 179fcddf-e247-4577-804b-267feed8abb1 = client
# (for 192.168.1.103_annex (rachel@rainbow.local:~/annex))
group 256d5762-150d-4d5d-9340-517de298c874 = client
# (for twilight.local_annex (rachel@twilight:~/annex))
group aeef7490-ce27-4255-b800-1947706c4a06 = client
# (for rachel@octavia:~/annex)
group c469fbce-f3b4-4e27-a54f-0b747797a7d5 = client
# (for annex (Biblio's Copy))
group c9e307e2-1189-47ed-8ad4-03b5c1b64e36 = client
# (for octavia.local_annex (rachel@octavia:~/annex))
group f748a5ed-d870-48fb-b3ec-811488eb2faa = client
# (for rachel@twilight:~/annex)
group fcaba03e-1ba5-11e3-90f1-57fe1467e006 = client
# (for luna.strangenoises.org_annex)
group f36dbdf8-1bba-11e3-9dbe-f33cfb0e2bed = transfer
# (for web)
#group 00000000-0000-0000-0000-000000000001 =

# Repository preferred contents
# (for rachel@octavia:~/annex)
content 161dec38-e8be-43b8-86c5-555d35ce3416 = standard
# (for rachel@celestia.local:~/annex)
content 179fcddf-e247-4577-804b-267feed8abb1 = standard
# (for 192.168.1.103_annex (rachel@rainbow.local:~/annex))
content 256d5762-150d-4d5d-9340-517de298c874 = standard
# (for twilight.local_annex (rachel@twilight:~/annex))
content aeef7490-ce27-4255-b800-1947706c4a06 = standard
# (for rachel@octavia:~/annex)
content c469fbce-f3b4-4e27-a54f-0b747797a7d5 = standard
# (for annex (Biblio's Copy))
content c9e307e2-1189-47ed-8ad4-03b5c1b64e36 = standard
# (for luna.strangenoises.org_annex)
content f36dbdf8-1bba-11e3-9dbe-f33cfb0e2bed = standard
# (for octavia.local_annex (rachel@octavia:~/annex))
content f748a5ed-d870-48fb-b3ec-811488eb2faa = standard
# (for rachel@twilight:~/annex)
content fcaba03e-1ba5-11e3-90f1-57fe1467e006 = standard
# (for web)
#content 00000000-0000-0000-0000-000000000001 =
"""]]

while the same command in /Volumes/Biblio/annex gives:

[[!format sh """
# git-annex configuration
#
# Changes saved to this file will be recorded in the git-annex branch.
#
# Lines in this file have the format:
#   setting uuid = value

# Repository trust configuration
# (Valid trust levels: trusted semitrusted untrusted dead)
# (for web)
#trust 00000000-0000-0000-0000-000000000001 = semitrusted
# (for rachel@octavia:~/annex)
#trust 161dec38-e8be-43b8-86c5-555d35ce3416 = semitrusted
# (for celestia.local (rachel@celestia.local:~/annex))
#trust 179fcddf-e247-4577-804b-267feed8abb1 = semitrusted
# (for rachel@rainbow.local:~/annex)
#trust 256d5762-150d-4d5d-9340-517de298c874 = semitrusted
# (for rachel@twilight:~/annex)
#trust aeef7490-ce27-4255-b800-1947706c4a06 = semitrusted
# (for rachel@octavia:~/annex)
#trust c469fbce-f3b4-4e27-a54f-0b747797a7d5 = semitrusted
# (for Biblio's Copy)
#trust c9e307e2-1189-47ed-8ad4-03b5c1b64e36 = semitrusted
# (for )
#trust f36dbdf8-1bba-11e3-9dbe-f33cfb0e2bed = semitrusted
# (for rachel@octavia:~/annex)
#trust f748a5ed-d870-48fb-b3ec-811488eb2faa = semitrusted
# (for rachel@twilight:~/annex)
#trust fcaba03e-1ba5-11e3-90f1-57fe1467e006 = semitrusted

# Repository groups
# (Standard groups: client transfer backup incrementalbackup smallarchive archive source manual public unwanted)
# (Separate group names with spaces)
# (for rachel@octavia:~/annex)
group 161dec38-e8be-43b8-86c5-555d35ce3416 = client
# (for celestia.local (rachel@celestia.local:~/annex))
group 179fcddf-e247-4577-804b-267feed8abb1 = client
# (for rachel@rainbow.local:~/annex)
group 256d5762-150d-4d5d-9340-517de298c874 = client
# (for rachel@twilight:~/annex)
group aeef7490-ce27-4255-b800-1947706c4a06 = client
# (for rachel@octavia:~/annex)
group c469fbce-f3b4-4e27-a54f-0b747797a7d5 = client
# (for Biblio's Copy)
group c9e307e2-1189-47ed-8ad4-03b5c1b64e36 = client
# (for rachel@octavia:~/annex)
group f748a5ed-d870-48fb-b3ec-811488eb2faa = client
# (for rachel@twilight:~/annex)
group fcaba03e-1ba5-11e3-90f1-57fe1467e006 = client
# (for )
group f36dbdf8-1bba-11e3-9dbe-f33cfb0e2bed = transfer
# (for web)
#group 00000000-0000-0000-0000-000000000001 =

# Repository preferred contents
# (for rachel@octavia:~/annex)
content 161dec38-e8be-43b8-86c5-555d35ce3416 = standard
# (for celestia.local (rachel@celestia.local:~/annex))
content 179fcddf-e247-4577-804b-267feed8abb1 = standard
# (for rachel@rainbow.local:~/annex)
content 256d5762-150d-4d5d-9340-517de298c874 = standard
# (for rachel@twilight:~/annex)
content aeef7490-ce27-4255-b800-1947706c4a06 = standard
# (for rachel@octavia:~/annex)
content c469fbce-f3b4-4e27-a54f-0b747797a7d5 = standard
# (for Biblio's Copy)
content c9e307e2-1189-47ed-8ad4-03b5c1b64e36 = standard
# (for )
content f36dbdf8-1bba-11e3-9dbe-f33cfb0e2bed = standard
# (for rachel@octavia:~/annex)
content f748a5ed-d870-48fb-b3ec-811488eb2faa = standard
# (for rachel@twilight:~/annex)
content fcaba03e-1ba5-11e3-90f1-57fe1467e006 = standard
# (for web)
#content 00000000-0000-0000-0000-000000000001 =
"""]]

### What steps will reproduce the problem?

As above. I have no idea what just happened, but apart from git-annex assistant --stop and having to mop up leftover processes, I didn't use the git-annex commandline for anything.

### What version of git-annex are you using? On what operating system?

Mac OS X 10.8.4

    Version: 4.20130909-ga29f960
    Build flags: Assistant Webapp Pairing Testsuite S3 WebDAV FsEvents XMPP DNS Feeds Quvi

### Please provide any additional information below.

The log on ~/annex/.git/annex/daemon.log is huge and full of transfers of files with my personal filenames. I'd rather not. It appears to end normally.

Now there is a short log in /Volumes/Biblio/annex/.git/annex/daemon.log from, I guess, the time I tried to restart. For some reason therefore, after the successful session finished, on restart it only looks here. This log is appended.

[[!format sh """
[2013-09-12 21:35:39 BST] main: starting assistant version 4.20130909-ga29f960

[2013-09-12 21:35:39 BST] TransferScanner: Syncing with celestia.local
Already up-to-date.

(scanning...) [2013-09-12 21:35:39 BST] Watcher: Performing startup scan
From /Users/rachel/annex
 * [new branch]      git-annex  -> celestia.local/git-annex
 * [new branch]      master     -> celestia.local/master
 * [new branch]      synced/git-annex -> celestia.local/synced/git-annex
 * [new branch]      synced/master -> celestia.local/synced/master
Updating 4f974a8..74770d9
Fast-forward
Already up-to-date.
Already up-to-date.
Already up-to-date.
[2013-09-12 21:36:39 BST] Pusher: Syncing with celestia.local
(merging celestia.local/git-annex celestia.local/synced/git-annex into git-annex...)
(Recording state in git...)

(started...) error: Ref refs/heads/synced/git-annex is at 5b4ed9b3098e936d60b61a1d3915fa29e8c823d0 but expected 792d2a5c14b0b6327d2089e174063c474ba5a764
remote: error: failed to lock refs/heads/synced/git-annex
To /Users/rachel/annex
   792d2a5..5b4ed9b  git-annex -> synced/git-annex
To /Users/rachel/annex
 ! [remote rejected] git-annex -> synced/git-annex (failed to lock)
error: failed to push some refs to '/Users/rachel/annex'
Everything up-to-date
"""]]

Well, I see that thing about "failed to lock". I can imagine that my 'killall git-annex' to kill a leftover process that was hanging around after I'd done git-annex assistant --stop might have left stale lock files, somewhere... but of course I only got as far as doing that because I was already encountering problems, just trying to return to the webapp.

> The original bug report seems to be a case of user confusion,
> and not a bug. (Although perhaps the UI is confusing?)
>
> The "resource exhausted" that came up later is quite likely the problem
> fixed in [[!commit 4d06037fdd44ba38fcd4c118d1e6330f06e22366]],
> which affected local git remotes.
>
> [[closing|done]]; I don't see any value keeping this open, I'm afraid.
> --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.51"
 subject="comment 1"
 date="2013-09-12T21:29:39Z"
 content="""
The git-annex webapp displays a view from \"inside\" one git repository at a time. It seems to me that you have two or more local git repositories; one in ~/annex and one on a removable drive. The webapp is coming up viewing from inside your removable drive's repository. You can change the repository that the webapp views by going to the Repository menu in the upper-right corner and selecting Switch repository. The webapp will remember which repository it viewed last, and come back up showing that same repository.

It's BTW an unusual configuration to have the git-annex assistant directly running on a removable drive as you seem to have done. You must have previously went to the Repository menu and selected \"Add another local repository\", and then added the removable drive, and then connected that repository up to your ~/annex repository. Which would explain why the webapp was last showing the repository on the removable drive. The more usual way to add a removable drive is to use the \"Add another repository\" button and select \"Removable drive\".

I think, but am not sure, that the error \"Ref refs/heads/synced/git-annex is at 5b4ed9b3098e936d60b61a1d3915fa29e8c823d0 but expected 792d2a5c14b0b6327d2089e174063c474ba5a764\" is due to having two git-annex assistants running in these two different repositories and both updating them both at the same time. You might want to stop all git-annex processes, edit `~/.config/git-annex/autostart` and remove the removable drive repository from that list, leaving only `~/annex` in the list.
"""]]

repo_name: kakafarm/git

license: unknown


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlEhzszkzOIy8-Rx8b2mcr75QcnIc6O_OA"
 nickname="Rachel"
 subject="I don't know whether I've fixed it or confused things further"
 date="2013-09-12T21:30:44Z"
 content="""
I shut down the daemon that was running from the webapp itself.

Instead of launching by using the app, I did:

[[!format txt \"\"\"
celestia:~ rachel$ cd annex/
celestia:annex rachel$ git-annex assistant
(merging synced/git-annex into git-annex...)
celestia:annex rachel$ git-annex webapp
Launching web browser on file:///Users/rachel/annex/.git/annex/webapp.html
celestia:annex rachel$
\"\"\"]]

It opened the webapp. I could see all the repos back again.

It also seems to be syncing to everything else, sending the files all over again, even though they haven't changed.

It's just about possible it's completing the job, as I'm not sure I saw the files syncing now, being synced earlier. But... in that earlier session it had run the queue dry, *it* thought it had completed syncing.

What's going on?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.51"
 subject="comment 3"
 date="2013-09-12T21:33:18Z"
 content="""
See my explanation above. By manually starting the webapp inside your ~/annex repository you forced it to view that repository.

It's not unusual for the webapp to display it syncing some files that have been synced before. This repository may not be up-to-date on which files have been sent where, and it will quickly notice the file has already been transferred and skip doing anything for that file.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlEhzszkzOIy8-Rx8b2mcr75QcnIc6O_OA"
 nickname="Rachel"
 subject="Last night's comment"
 date="2013-09-13T10:28:02Z"
 content="""
> It's BTW an unusual configuration to have the git-annex assistant directly running on a removable drive as you seem to have done. You must have previously went to the Repository menu and selected \"Add another local repository\", and then added the removable drive, and then connected that repository up to your ~/annex repository. Which would explain why the webapp was last showing the repository on the removable drive. The more usual way to add a removable drive is to use the \"Add another repository\" button and select \"Removable drive\".

That is exactly how I'd added that repository, yes. I didn't notice the dedicated \"removable drive\" option until later. :-) I thought it possibly appropriate to leave it that way because, while Biblio is an external drive and remov*able*, in practice I never remove it; it's a permanent fixture.

After all, this is a mac mini server, originally (no longer running Server) so it has a second *internal* drive. It would have been even more logical for me to have added a second repo on there in the same manner, as it's most definitely not removable without extreme effort; but it still mounts inside /Volumes, and presumably if I'd added it as a non-removable repo, I'd have had the same problem?

> It's not unusual for the webapp to display it syncing some files that have been synced before. This repository may not be up-to-date on which files have been sent where, and it will quickly notice the file has already been transferred and skip doing anything for that file.

Yes, that's probably what happened. It finished pretty quickly, about a minute after sending my previous comment.

Going to bed now, but tomorrow I'll follow the remaining suggestions; basically I'll remove the repo on the external drive then, if I still want it there, I'll add it back in the other way, so it thinks of it as removable.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlEhzszkzOIy8-Rx8b2mcr75QcnIc6O_OA"
 nickname="Rachel"
 subject="And this morning..."
 date="2013-09-13T10:38:35Z"
 content="""
I tried removing the second repo, by deleting it, it seemed to begin normally. Then it just hung, and hung.

Looking at logs, it seemed to be hung on dropping one file - a very small file in fact, but it was probably the most recent file to be added. So the end of the log just looked like:

[[!format txt \"\"\"
drop annex old stuff/renegade/issue7back.pdf ok
drop annex old stuff/renegade/issue7fiction.pdf ok
drop annex postgresql 9.0->9.1 upgrade process
\"\"\"]]

After a while I started getting problems in other terminal shells where I was doing stuff unrelated to git annex. eg: on trying to open a new terminal window, it would open, then shut immediately, or it would open with just:

[[!format txt \"\"\"
forkpty: Resource temporarily unavailable
Could not create a new process and open a pseudo-tty.
\"\"\"]]

Various other commands that would have resulted in a fork were failing too. In the end I shut down git annex (using the shutdown daemon open in the menu in the webapp), and everything went back to normal.

This is the end of the daemon.log from this morning. I don't want to paste the whole thing, as essentially it lists all my private filenames, and there's a lot. In fact I wonder if the quantity of files may be a factor:

[[!format txt \"\"\"
drop annex old stuff/renegade/index.html ok
drop annex old stuff/renegade/issue1.pdf ok
drop annex old stuff/renegade/issue2.pdf ok
drop annex old stuff/renegade/issue3.pdf ok
drop annex old stuff/renegade/issue4.pdf ok
drop annex old stuff/renegade/issue4coverpic.pdf ok
drop annex old stuff/renegade/issue6articles.pdf ok
drop annex old stuff/renegade/issue6cover.pdf ok
drop annex old stuff/renegade/issue6fiction.pdf ok
drop annex old stuff/renegade/issue7articles.pdf ok
drop annex old stuff/renegade/issue7back.pdf ok
drop annex old stuff/renegade/issue7fiction.pdf ok
drop annex postgresql 9.0->9.1 upgrade process [2013-09-13 11:25:07 BST] NetWatcherFallback: Syncing with twilight.local_annex, octavia.local_annex, 192.168.1.103_annex, luna.strangenoises.org_annex
NetWatcherFallback crashed: git: createProcess: resource exhausted (Resource temporarily unavailable)
[2013-09-13 11:25:07 BST] NetWatcherFallback: warning NetWatcherFallback crashed: git: createProcess: resource exhausted (Resource temporarily unavailable)
recv: resource vanisrhreeecdcv v:(: C rorenesnsoeoucurtrciceoe n v varanenisiseshthe edbd y ( (CpCoeonennrne)ec
cttiioonn  rreesseett  bbyy  ppeeeerr))

[2013-09-13 11:26:32 BST] main: warning git-annex has been shut down
\"\"\"]]

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlEhzszkzOIy8-Rx8b2mcr75QcnIc6O_OA"
 nickname="Rachel"
 subject="comment 6"
 date="2013-09-13T10:52:07Z"
 content="""
FYI, if there's any relevance to the number of files in the annex, there are 1899 files in the annex at the moment, so that many in the one being deleted. The one it hung on, \"postgresql 9.0->9.1 upgrade process\" was indeed the last one that comes up in a find command, which I think means the most recently added to this dir.

Creating too many threads?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlEhzszkzOIy8-Rx8b2mcr75QcnIc6O_OA"
 nickname="Rachel"
 subject="comment 7"
 date="2013-09-13T10:54:37Z"
 content="""
... and resolved for myself by just deleting (dropping in the trash for now) the annex on the external volume, restarting git-annex assistant and *disabling* that repo from the menu rather than deleting it.

"""]]


code:
### Please describe the problem.
On the first run it seems "Open Webapp" always use the URL used to configure a new repo.

### What steps will reproduce the problem?
I'm using a /sdcard/DCIM annex.

1. close the tabs in the git annex terminal to close git annex
2. force close the application in the settings and delete data (not sure if it's needed)
3. delete /sdcard/DCIM/.git/ and /sdcard/git-annex.home/ (I'm not sure I'm deleting everything, maybe I missed some dirs)
4. start git annex

(WebApp opens and ask to create a repo) see: [http://i.imgur.com/v0EUuir.png](http://i.imgur.com/v0EUuir.png)

5. Click "Make Camera Repository"

(WebApp scans and commits the files) see: [http://i.imgur.com/97rYfXa.png](http://i.imgur.com/97rYfXa.png) (note that the port changed)

6. Return to the terminal. see: [http://i.imgur.com/edL5and.png](http://i.imgur.com/edL5and.png)
7. Open WebApp using the menu

(Git annex opens the URL to configure a new repo (the webapp is still accessible with the other url). see: [http://i.imgur.com/T8A5NTD.png](http://i.imgur.com/T8A5NTD.png)

If I close and reopen the terminal, the good url is used when I use "open webapp"

### What version of git-annex are you using? On what operating system?
4.20130601-g7483ca4

### Please provide any additional information below.

[[!format sh """
[2013-06-01 17:13:34 EDT] main: starting assistant version 4.20130601-g7483ca4
(scanning...) [2013-06-01 17:13:34 EDT] Watcher: Performing startup scan
(started...) [2013-06-01 17:13:42 EDT] Committer: Adding 123 files

add Camera/IMG_20130223_111411.jpg (checksum...) ok
[...]
[2013-06-01 17:14:17 EDT] Committer: Committing changes to git
"""]]

> [[fixed|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-06-01T23:39:39Z"
 content="""
Ah yes, I see what's going on. It's not updating the url in the special file added for Android.

Should be an easy fix one I return from vacation.

Workaround: Open a new terminal tab and it will open the web browser viewing the repo you set up.
"""]]


code:
Before I start on what's gone wrong, many thanks for a great program: finally a way of finding out where I've put all those files, and I enjoyed your talk in Australia. Not quite to New Zealand, but a good start :-)

To play with git-annex, I decided to convert my ~/Downloads directory to a git-annex repository, as there were a wide variety of files in it, mostly easily replaceable, and also handy to have on multiple machines. In hindsight, probably wasn't a great idea, as I'd regularly forget it was a git-annex repo and move files and directories out manually, which caused all sorts of fun trying to sort out the dead symlinks. Then I after one update, the repository format changed, from WORM to xxx256 format which meant there were now both sorts in the GA object store.

More recently I'd tried converting the repo to direct mode, you get the idea: lots of playing with git-annex commands, and now that I think about it, possibly some git commands too trying to repair missing files.

Anyway I've ended up with a 27GB git-annex repo that now manages to kill git-annex whenever I try to check it using "git-annex fsck".

Not only does the fsck subcommand cause it to die, but also "find", "whereis" and "status". It dies on the same file (for find/whereis/fsck).

e.g.

    ... lots of stuff deleted ...
    whereis 1wolf14.zip (2 copies)
            051f0b00-e265-11e1-894e-3b0b3f3844f2 -- Laptop
            2c4e11e0-a1b4-11e1-9a02-73e17b04c00f -- here (myPC - Downloads)
    ok
    git-annex: out of memory (requested 2097152 bytes)

Now "git status" for some repo data:

    myPC:~/Downloads$ git annex status
    supported backends: SHA256E SHA1E SHA512E SHA224E SHA384E SHA256 SHA1 SHA512 SHA224 SHA384 WORM URL
    supported remote types: git S3 bup directory rsync web webdav glacier hook
    repository mode: direct
    trusted repositories: 0
    semitrusted repositories: 4
            00000000-0000-0000-0000-000000000001 -- web
            051f0b00-e265-11e1-894e-3b0b3f3844f2 -- Laptop
            2c4e11e0-a1b4-11e1-9a02-73e17b04c00f -- here (myPC - Downloads)
            48fbe52a-a1b3-11e1-bb80-ebc15118871d -- netbk
    untrusted repositories: 0
    dead repositories: 0
    transfers in progress: none
    available local disk space: 100 gigabytes (+1 megabyte reserved)
    local annex keys: 1719
    local annex size: 27 gigabytes
    known annex keys: git-annex: out of memory (requested 2097152 bytes)

It always seems to die at about 3.5GB memory usage. This is running on Ubuntu 12.04, using the latest GA release built using cabal:

    git-annex version: 4.20130227
    local repository version: 3
    default repository version: 3
    supported repository versions: 3 4
    upgrade supported from repository versions: 0 1 2

There are also dead symlinks that point to directories that have meta-data but not the symlink target (manually line-wrapped):

    myPC:~/Downloads$ ls -l precise*
    lrwxrwxrwx 1 nino nino 194 Oct 21 12:48 precise-dvd-i386.iso ->
                 .git/annex/objects/0x/Xz/SHA256-s3590631424--
                 b08ecdd4846948ec076b23afae7f87be9cfba5218fb9ba4160f26c0b8d4b5dd0/
                 SHA256-s3590631424--b08ecdd4846948ec076b23afae7f87be9cfba5218fb9ba4160f26c0b8d4b5dd0

But looking in the symlink destination directory, there's no corresponding object, only metadata:

    myPC:~/Downloads/.git/annex/objects/0x/Xz/SHA256-s3590631424--b08ecdd4846948ec076b23afae7f87be9cfba5218fb9ba4160f26c0b8d4b5dd0$ ls -l
    total 8
    -rw-rw-r-- 1 nino nino 30 Jan  8 23:15 SHA256-s3590631424--b08ecdd4846948ec076b23afae7f87be9cfba5218fb9ba4160f26c0b8d4b5dd0.cache
    -rw-rw-r-- 1 nino nino 49 Jan  8 23:15 SHA256-s3590631424--b08ecdd4846948ec076b23afae7f87be9cfba5218fb9ba4160f26c0b8d4b5dd0.map

But there is another version somewhere else.

    -r--r--r-- 1 nino nino 3590631424 Mar 18  2012 ./.git/annex/objects/Kj/wM/WORM-s3590631424-m1331991509
                        --precise-dvd-i386.iso/WORM-s3590631424-m1331991509--precise-dvd-i386.iso

This actual file does exist in the "Used" directory:

    -rw-r--r-- 1 nino nino 3.4G May 27  2012 precise-dvd-i386.iso

I'm not so worried about the mangled repo - it's quite possibly because of clueless git/git-annex command usage - but the inability to use the fsck command is concerning

I could just uninit everything, but as it dies prematurely, I'm not certain that all the contents would be restored.
Any thoughts on how I can get git-annex (esp. fsck) to complete would be appreciated.

Thanks
Giovanni

> [[fixed|done]]. However, if you saw this behavior,
> you have large files checked directly into git. You may
> want to examine your repository and use git filter-branch to clean
> it up.
> --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-03-03T18:34:41Z"
 content="""
git-annex is designed to run in constant memory. It should never use a lot of memory.

What file causes the problem? You show git-annex whereis successfully processing 1wolf14.zip, so that file is apparently not the problem. It's probably the *next* file you told whereis to run on. You can get a file list with `git ls-files`

Apparently something bad is happening when it tries to calculate the key used by the problem file.

What does the file look like? You're using direct mode so it may or may not be a symlink. If it's a symlink, show it. If not, show the output of: `echo HEAD:./$filename | git cat-file --batch`

----

The stuff about \"dead symlinks\" etc is all a distraction AFAICS. git-annex allows you to remove content from your repository in multiple ways. That content could be present in another repository (git annex whereis would tell you), or the last copy of it could have been deleted. You can run \"git annex log\" on a file to show where the content was located historically.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 2"
 date="2013-03-18T23:25:24Z"
 content="""
This is an important bug. Could you please get back to me so I can debug it?
"""]]


code:
[[!comment format=mdwn
 username="Xyem"
 ip="87.194.19.134"
 subject="comment 3"
 date="2013-03-22T11:18:53Z"
 content="""
I may have a bug along the same lines of this but I think it is in git, rather than git-annex.

I've been using git-annex to sort and consolidate ~2TB files that have been spread across several machines. I probably failed to do an important step (such as committing to git, I'm not sure) while moving a lot of these files around which renders me unable to use the repository.

For example, if I run any command that uses something like 'git status' in the background, it gradually consumes all available memory. I've not had an OOM error yet (8GB RAM, 2GB swap) but it also never seems to complete.

I just did a quick test on this and running 'git status' in the annex results in, after a few minutes, a growing number of 'git status --porcelain' processes (quickly reaches 70+, currently at 94) and nearly all memory consumed. This would normally be triggered with 'git annex sync' or some other command which uses 'git status' to do some work.

Hope this information is of some use. If I can help further, please let me know.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 4"
 date="2013-03-27T16:38:07Z"
 content="""
@Xyem, sounds like an unrelated bug. Also, no git-annex commands run `git status`. You can find other git commands that may be using a lot of memory by passing the --debug flag to git-annex.
"""]]


code:
[[!comment format=mdwn
 username="Xyem"
 ip="87.194.19.134"
 subject="comment 5"
 date="2013-03-27T16:50:58Z"
 content="""
@joey: Hm.. I'm sure it was 'git annex sync' that initially consumed all RAM by spawning a lot of 'git status' processes. However, I can't replicate it anymore as I found a way to make the repository useable again.

I went low into the directory structure and added/committed from there. Now I can 'git annex sync' and 'git status' from the annex root without any problems.

Sorry for cluttering this bug report up, thought it might be related/of some help.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.14.105"
 subject="comment 7"
 date="2013-09-19T19:07:03Z"
 content="""
[[Huge_annex_out_of_memory_on_switch_to_indirect_mode_and_status]] is a duplicate of this bug.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlVUq_c3-lrQBculOEUu3yjvdavE7JbvEI"
 nickname="Stig"
 subject="comment 8"
 date="2014-07-25T10:02:53Z"
 content="""
\"git annex sync\" or \"git annex fsck\" gives me the same problem.  This is an annex which has previously been running with the git annex assistant.

Output is first:

    $ git annex sync
    (merging synced/git-annex into git-annex...)

…then the workstation starts swapping, and after eating 16 GB RAM, and all of 16 GB swap, OOM killer komes for \"git\".

The process which eats up all the memory is:

    git --git-dir=/home/ssm/annex/.git --work-tree=/home/ssm/annex \
      -c core.bare=false log \
      7bed443dc22961214f86e65aedb8861affd215d3..refs/heads/git-annex \
      -n1 --pretty=%H

I _think_ that since a \"-n1\" argument is given, it will only show the log for the last commit in the range, and one could specify \"refs/heads/git-annex\" instead of the range.  With just \"refs/heads/git-annex\" instead of the range, it returns a reference instantly.

The output of git count-objects is

    $ git count-objects -H -v
    count: 334758
    size: 6.27 GiB
    in-pack: 16600
    packs: 1
    size-pack: 1.70 MiB
    prune-packable: 0
    garbage: 0
    size-garbage: 0 bytes

…and there is 1043 files in the annex.
"""]]


code:
Open two webapp sessions (in two different *screen* windows or whatever)

    name@name-Computermodel-1000:~/test/annex1$ git-annex webapp
    name@name-Computermodel-1000:~/test/annex2$ git-annex webapp

In annex1's web UI, choose *Configuration* -> *Manage repositories* -> *Local computer*.

Fill in the secret phrase and press *Start pairing*.

In annex2's web UI, a pairing request will show on the left notifications bar. Choose *Respond*, fill in the phrase field and press *Finish pairing*.

I was expecting a pairing between the two running session. But I got this after pressing *Finish pairing* in annex2's web UI:

**start of output**
# Internal Server Error

bad comment in ssh public key ssh-rsa [very long GPG key jibber jabber] name@name-Computermodel-1000

**end of output**

git-annex version: 3.20121017, Ubuntu 12.04

> [[Done]], allowed dash and underscore in there now.


code:
I get the following error message upon starting git-annex in a second user account on Android:

    Falling back to hardcoded app location: cannot find expected files in /data/app-lib
    git annex webapp
    lib/lib.runshell.so: line 133: git: Permission denied

    [Terminal session finished]

The same version of git-annex works just fine for the primary user.
(The primary user has root access which unfortunately can't be enabled for other user accounts.)

### What version of git-annex are you using? On what operating system?

  * git-annex: 5.20140710
  * OS: CyanogenMod 10.1.3-p3110


code:
With the storage layout v3, pulling the git-annex branch into the master branch is... less than ideal.

The fact that the two branches contain totally different data make an accidental merge worse, arguably.

Adding a tiny binary file called .gitnomerge to both branches would solve that without any noticeable overhead.

Yes, there is an argument to be made that this is too much hand-holding, but I still think it's worth it.

-- Richard

> It should be as easy to undo such an accidential merge
> as it is to undo any other git commit, right? I quite like that git-annex
> no longer adds any clutter to the master branch, and would be reluctant
> to change that. --[[Joey]]

[[wontfix|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 1"
 date="2011-10-23T15:00:48Z"
 content="""
Having run into the same issue again, I still think git-annex should ensure no merges take place. The clutter introduced by a .gitnomerge is neglible, imo.
"""]]


code:
What steps will reproduce the problem?

Adding a e.g box.com repository from behind a http proxy via webapp.

What is the expected output? What do you see instead?

Connection should be made. But there is an error message:

"Internal Server Error
connect: does not exist (Connection refused): user error"

What version of git-annex are you using? On what operating system?

3.20121127 on Archlinux

Please provide any additional information below.

I don't use networkmanager if proxy information is obtained from it. There should be a fallback to environment variables.
[[!tag confirmed]]


code:
Addurl can fail due to an apparent race condition when watch or assistant is running and the repository is in direct mode. The following stress test script encounters the bug consistently on my system. I am running git-annex 5.20140320 on on Ubuntu 13.10.

[[!format sh """
#!/bin/sh
set -eu

cleanup() {
  local dir
  dir="$1"; shift
  if [ -d "$dir" ]; then
    (
      set -x
      fuser -k -w "$dir/annex/.git/annex/daemon.log" || :
      find "$dir" -type d -exec chmod 700 '{}' '+'
      find "$dir" -type f -exec chmod 600 '{}' '+'
      rm -fr "$dir"
    )
  fi
}

go() {
  local dir
  dir="$(mktemp -d "${TMP:-/tmp}/stress-annex.XXXXXXXXXX")"
  trap "cleanup '$dir'" 0 1 2 13 15

  (
    cd "$dir"
    mkdir annex
    cd annex
    set -x

    git init
    git annex init
    git annex direct
    git annex watch

    for n in $(seq 100); do
      git annex addurl --file=foo http://heh.fi/robots.txt
      git annex sync
      rm -f foo
      git annex sync
    done

    git annex watch --stop
    git annex uninit
  )

  cleanup "$dir"
  trap - 0 1 2 13 14
}

go
"""]]

Script output:

[[!format sh """
% ./stress-annex
+ git init
Initialized empty Git repository in /tmp/stress-annex.OKj6D8kVmV/annex/.git/
+ git annex init
init  ok
(Recording state in git...)
+ git annex direct
commit
On branch master

Initial commit

nothing to commit
ok
direct  ok
+ git annex watch
+ seq 100
+ git annex addurl --file=foo http://heh.fi/robots.txt
addurl foo (downloading http://heh.fi/robots.txt ...)
--2014-03-27 03:14:29--  http://heh.fi/robots.txt
Resolving heh.fi (heh.fi)... 83.145.237.222
Connecting to heh.fi (heh.fi)|83.145.237.222|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0 [text/plain]
Saving to: ‘/tmp/stress-annex.OKj6D8kVmV/annex/.git/annex/tmp/URL--http&c%%heh.fi%robots.txt’

    [ <=>                                                                                ] 0           --.-K/s   in 0s

2014-03-27 03:14:29 (0.00 B/s) - ‘/tmp/stress-annex.OKj6D8kVmV/annex/.git/annex/tmp/URL--http&c%%heh.fi%robots.txt’ saved [0/0]

(Recording state in git...)
ok
(Recording state in git...)
+ git annex sync
commit  ok
+ rm -f foo
+ git annex sync
commit  (Recording state in git...)
ok
(Recording state in git...)
+ git annex addurl --file=foo http://heh.fi/robots.txt
addurl foo (downloading http://heh.fi/robots.txt ...)
--2014-03-27 03:14:29--  http://heh.fi/robots.txt
Resolving heh.fi (heh.fi)... 83.145.237.222
Connecting to heh.fi (heh.fi)|83.145.237.222|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0 [text/plain]
Saving to: ‘/tmp/stress-annex.OKj6D8kVmV/annex/.git/annex/tmp/URL--http&c%%heh.fi%robots.txt’

    [ <=>                                                                                ] 0           --.-K/s   in 0s

2014-03-27 03:14:29 (0.00 B/s) - ‘/tmp/stress-annex.OKj6D8kVmV/annex/.git/annex/tmp/URL--http&c%%heh.fi%robots.txt’ saved [0/0]

(Recording state in git...)
ok
(Recording state in git...)
+ git annex sync
commit  ok
+ rm -f foo
+ git annex sync
commit  (Recording state in git...)
ok
(Recording state in git...)
+ git annex addurl --file=foo http://heh.fi/robots.txt
addurl foo (downloading http://heh.fi/robots.txt ...)
--2014-03-27 03:14:29--  http://heh.fi/robots.txt
Resolving heh.fi (heh.fi)... 83.145.237.222
Connecting to heh.fi (heh.fi)|83.145.237.222|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0 [text/plain]
Saving to: ‘/tmp/stress-annex.OKj6D8kVmV/annex/.git/annex/tmp/URL--http&c%%heh.fi%robots.txt’

    [ <=>                                                                                ] 0           --.-K/s   in 0s

2014-03-27 03:14:29 (0.00 B/s) - ‘/tmp/stress-annex.OKj6D8kVmV/annex/.git/annex/tmp/URL--http&c%%heh.fi%robots.txt’ saved [0/0]

git-annex: /tmp/stress-annex.OKj6D8kVmV/annex/.git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/: openTempFile: permission denied (Permission denied)
failed
git-annex: addurl: 1 failed
+ fuser -k -w /tmp/stress-annex.OKj6D8kVmV/annex/.git/annex/daemon.log
/tmp/stress-annex.OKj6D8kVmV/annex/.git/annex/daemon.log: 30704 30709 30735 30738 30778
+ find /tmp/stress-annex.OKj6D8kVmV -type d -exec chmod 700 {} +
+ find /tmp/stress-annex.OKj6D8kVmV -type f -exec chmod 600 {} +
+ rm -fr /tmp/stress-annex.OKj6D8kVmV
"""]]

The script also seems to encounter another issue. The output when seq 100 is changed to seq 1 and addurl happens to succeed:

[[!format sh """
+ git init
Initialized empty Git repository in /tmp/stress-annex.QEs0pNyS9z/annex/.git/
+ git annex init
init  ok
(Recording state in git...)
+ git annex direct
commit
On branch master

Initial commit

nothing to commit
ok
direct  ok
+ git annex watch
+ seq 1
+ git annex addurl --file=foo http://heh.fi/robots.txt
addurl foo (downloading http://heh.fi/robots.txt ...)
--2014-03-27 03:17:20--  http://heh.fi/robots.txt
Resolving heh.fi (heh.fi)... 83.145.237.222
Connecting to heh.fi (heh.fi)|83.145.237.222|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0 [text/plain]
Saving to: ‘/tmp/stress-annex.QEs0pNyS9z/annex/.git/annex/tmp/URL--http&c%%heh.fi%robots.txt’

    [ <=>                                                                                ] 0           --.-K/s   in 0s

2014-03-27 03:17:20 (0.00 B/s) - ‘/tmp/stress-annex.QEs0pNyS9z/annex/.git/annex/tmp/URL--http&c%%heh.fi%robots.txt’ saved [0/0]

(Recording state in git...)
ok
(Recording state in git...)
+ git annex sync
commit  ok
+ rm -f foo
+ git annex sync
commit  (Recording state in git...)
ok
(Recording state in git...)
+ git annex watch --stop
+ git annex uninit
git-annex: /tmp/stress-annex.QEs0pNyS9z/annex/.git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.map: removeLink: permission denied (Permission denied)
+ fuser -k -w /tmp/stress-annex.QEs0pNyS9z/annex/.git/annex/daemon.log
+ :
+ find /tmp/stress-annex.QEs0pNyS9z -type d -exec chmod 700 {} +
+ find /tmp/stress-annex.QEs0pNyS9z -type f -exec chmod 600 {} +
+ rm -fr /tmp/stress-annex.QEs0pNyS9z
"""]]


code:
[[!comment format=mdwn
 username="http://johan.kiviniemi.name/"
 nickname="Johan"
 subject="Another race condition"
 date="2014-03-31T03:42:42Z"
 content="""
Here’s another race condition which seems related:

[[!format sh \"\"\"
% git annex addurl 'quvi:http://youtu.be/-CbFj9K9AQg'
addurl David_Raymond_Christiansen___Dependently_Typed_Programming_in_Idris___A_Demo.webm
--2014-03-31 05:45:49--  http://r4---sn-oxc0a5-ixae.googlevideo.com/videoplayback?<cut>
Resolving r4---sn-oxc0a5-ixae.googlevideo.com (r4---sn-oxc0a5-ixae.googlevideo.com)... 83.145.196.143, 2001:1bc8:100:1b::f
Connecting to r4---sn-oxc0a5-ixae.googlevideo.com (r4---sn-oxc0a5-ixae.googlevideo.com)|83.145.196.143|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://r13---sn-5go7dn7s.googlevideo.com/videoplayback?<cut> [following]
--2014-03-31 05:45:49--  http://r13---sn-5go7dn7s.googlevideo.com/videoplayback?<cut>
Resolving r13---sn-5go7dn7s.googlevideo.com (r13---sn-5go7dn7s.googlevideo.com)... 173.194.48.18, 2a00:1450:400f::12
Connecting to r13---sn-5go7dn7s.googlevideo.com (r13---sn-5go7dn7s.googlevideo.com)|173.194.48.18|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 320557466 (306M) [video/webm]
Saving to: ‘/home/ion/nobackup/media/video/.git/annex/tmp/URL--quvi&chttp&c%%youtu.be%-CbFj9K9AQg’

100%[======================================>] 320,557,466  895KB/s   in 6m 14s

2014-03-31 05:52:04 (837 KB/s) - ‘/home/ion/nobackup/media/video/.git/annex/tmp/URL--quvi&chttp&c%%youtu.be%-CbFj9K9AQg’ saved [320557466/320557466]

(Recording state in git...)
fatal: Unable to create '/home/ion/nobackup/media/video/.git/index.lock': File exists.

If no other git process is currently running, this probably means a
git process crashed in this repository earlier. Make sure no other git
process is running and remove the file manually to continue.

git-annex: user error (xargs [\"-0\",\"git\",\"--git-dir=/home/ion/nobackup/media/video/.git\",\"--work-tree=/home/ion/nobackup/media/video\",\"-c\",\"core.bare=false\",\"add\",\"--\"] exited 123)
failed
git-annex: addurl: 1 failed

% ls -l /home/ion/nobackup/media/video/.git/index.lock
ls: cannot access /home/ion/nobackup/media/video/.git/index.lock: No such file or directory
\"\"\"]]

The only lines in daemon.log from that time:

[[!format sh \"\"\"
[2014-03-31 05:52:04 EEST] Committer: Committing changes to git
[2014-03-31 05:52:04 EEST] Pusher: Syncing with heh.fi
Already up-to-date.
To heh.fi:/storage/ion/media/video
   3df241b..095d6c3  git-annex -> synced/git-annex
   10b3166..98074c1  annex/direct/master -> synced/master
Already up-to-date.
\"\"\"]]

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.244"
 subject="comment 2"
 date="2014-04-02T20:17:19Z"
 content="""
These races look beniegn, as far as I can see it doesn't cause any data to be lost, or indeed anything to happen that wouldn't have happened if addurl had been run without the assistant running.

The first race probably has addurl and the assistant both trying to move the file object into the annex at the same time. One wins and moves it; the other loses and sulks.

The second race has addurl trying to `git add` the file, while the assistant has already noticed the file appeared, `git add`ed it, and committed the add.

The only way to really avoid these races would be to add a lot of lock checking. Or just make `git annex addurl` and presumably also `git annex add` and maybe several other commands refuse to run when the assistant is running.
"""]]


code:
### Please describe the problem.
My remote repository's annex was lost.  In the webapp, I can now click to recreate it, that fails half-way.

### What steps will reproduce the problem?
Create a remote repo, delete the annex directory.

### What version of git-annex are you using? On what operating system?
git-annex version: 5.20140517-g0aed6d9
build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV FsEvents XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier ddar hook external
local repository version: unknown
supported repository version: 5
upgrade supported from repository versions: 0 1 2 4

### Please provide any additional information below.

[[!format sh """
gcrypt: Development version -- Repository format MAY CHANGE
gcrypt: Repository not found: ssh://lhunath@satura.lyndir.com/~/annex/
gcrypt: Setting up new repository
gcrypt: Remote ID is :id:ROTs3SImZEdvdO2yCMku
gcrypt: Encrypting to:  -R 0EF21226F43EA6BC
gcrypt: Requesting manifest signature
Warning: remote port forwarding failed for listen port 2222
Warning: remote port forwarding failed for listen port 37218
fatal: '~/annex/' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
"""]]

[[!tag moreinfo]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmZilYULa6CDEGfuagoDlesyakBgnf-dF8"
 nickname="Maarten"
 subject="comment 1"
 date="2014-05-21T22:21:37Z"
 content="""
I'd delete the repo and recreate it but I can't do that either - deleting works from webapp but when recreating it says there is already a repo by that name.  enableing that repo leads me to this situation again.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="24.159.78.125"
 subject="comment 2"
 date="2014-05-23T14:27:01Z"
 content="""
What are you clicking on in the webapp?

What kind of repository did the remote have before it got deleted? Was it a repository encrypted with git-remote-gcrypt?

Show the contents of ~/annex/.git/config on your local system.
"""]]


code:
What steps will reproduce the problem?

Create a new remote repository in the webapp. Get to the final phase of the setup where it asks you if you want to encrypt it, yet no other option is given to continue.

What is the expected output? What do you see instead?

At least two options:

1. Use an encrypted rsync repository on the server (the existing one)
2. Use an unencrypted rsync repository on the server

What version of git-annex are you using? On what operating system?

    $ ./git-annex version
    git-annex version: 3.20130102

    $ uname -a
    Linux wintermute 3.2.0-35-generic #55-Ubuntu SMP Wed Dec 5 17:45:18 UTC 2012 i686 i686 i386 GNU/Linux

    $ lsb_release -a
    Distributor ID:	Ubuntu
    Description:	Ubuntu 12.04.1 LTS
    Release:	12.04
    Codename:	precise

[[!meta title="webapp does not allow disabling encryption on rsync special remotes"]]
[[!tag /design/assistant]]
[[!tag confirmed]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.211"
 subject="comment 1"
 date="2013-01-03T18:02:21Z"
 content="""
Specifically, this affects only rsync special remotes. All the other ones have a check box that allows enabling encryption or not. I didn't get around to adding that for rsync due to some complications in the code. Of course, you can use `git annex initremote` at the command line to set up non-encrypted rsync remotes.
"""]]


code:
Setup:

* fresh install of Debian Wheezy with git-annex 4.20130227 pulled in from unstable
* clone existing repository and activate assistant
* repository has encrypted rsync remote originally setup with the name `metaarray`
* this remote was renamed to `ma` a long time ago, using the webapp
* had to perform this rename on each client

Steps:

* attempt to reactivate special remote using webapp repositories page, on reinstalled machine

Expected:

* special remote starts working
* renaming special remotes ought to survive clones

Actual:

* firstly, special remote activation page has blank hostname box and the hostname of the machine is in the username box
* form gives error "cannot change encryption type of existing remote"

Workaround:

* execute `git annex initremote metaarray`
* rename `metaarray` to `ma` again using the webapp

Perhaps the renaming of the remote not surviving clones is unavoidable, but the webapp should be able to cope with the situation.  Thanks.

[[!tag /design/assistant]]


code:
### Please describe the problem.

Clone of a repository does not contain the names of other repositories.

### What steps will reproduce the problem?

### What version of git-annex are you using? On what operating system?

Mac OS X

git-annex version: 5.20140308-g06fb279
build flags: Assistant Webapp Pairing Testsuite S3 WebDAV FsEvents XMPP DNS Feeds Qu
vi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256
 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier hook extern
al

### Please provide any additional information below.

Clones of my repositories lost all track of other repositories they only seem to know about them selfs attempting to remove other repositories (git annex dead name-of-an-existing-repo) fails, telling me that the repository with the name is not found.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

# End of transcript or log.
"""]]

[[!tag moreinfo]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.244"
 subject="comment 1"
 date="2014-04-07T19:24:12Z"
 content="""
Have you run `git-annex repair` or used the assistant to repair this repository? This can result in data going missing until the repository is able to pull the data from one of its remotes.
"""]]


code:
### Please describe the problem.

trying to add a remote host using its avahi local name : nas.local (for example).

### What steps will reproduce the problem?

add remote server, use nas.local > cannot resolve nas.local

### What version of git-annex are you using? On what operating system?

Version: 5.20140116-g2d9ec29
Ubuntu

### Please provide any additional information below.

> [[closing|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.68"
 subject="comment 1"
 date="2014-01-19T22:43:10Z"
 content="""
AFAIK, this will work fine if your OS is able to resolve those addresses using its regular resolver. Works fine here. Is `libnss-mdns` installed?
"""]]


code:
What steps will reproduce the problem?
My annex dir has 23459 files and uses 749MB disk space.
Just create a repository put this dir inside, and git-annex will crash.

What is the expected output? What do you see instead?
I expect git-annex handles large number of files, and does not watch every single file of it.

What version of git-annex are you using? On what operating system?
I'm using git-annex linux build, version 2013.04.17.

Please provide any additional information below.

    [2013-04-17 23:52:35 CEST] Transferrer: Downloaded pappas_hu..di_44.jpg
    git-annex: runInteractiveProcess: pipe: Too many open files
    Committer crashed: lsof: createProcess: resource exhausted (Too many open files)
    [2013-04-17 23:53:52 CEST] Committer: warning Committer crashed: lsof: createProcess: resource exhausted (Too many open files)
    git-annex: runInteractiveProcess: pipe: Too many open files
    git: createProcess: resource exhausted (Too many open files)
    DaemonStatus crashed: /home/user/Desktop/down/annex_test/.git/annex/daemon.status.tmp21215: openFile: resource exhausted (Too many open files)
    [2013-04-17 23:57:24 CEST] DaemonStatus: warning DaemonStatus crashed: /home/user/Desktop/down/annex_test/.git/annex/daemon.status.tmp21215: openFile: resource exhausted (Too many open files)
    git-annex: runInteractiveProcess: pipe: Too many open files
    git: createProcess: resource exhausted (Too many open files)
    git-annex: runInteractiveProcess: pipe: Too many open files
    NetWatcherFallback crashed: git: createProcess: resource exhausted (Too many open files)
    [2013-04-18 00:27:17 CEST] NetWatcherFallback: warning NetWatcherFallback crashed: git: createProcess: resource exhausted (Too many open files)
    git-annex: runInteractiveProcess: pipe: Too many open files
    git-annex: git: createProcess: resource exhausted (Too many open files)
    git-annex: accept: resource exhausted (Too many open files)

Instead of raising system's limit (which is a neverending story), can we make git-annex only watch a directory and not every file of it?

Or could the user specify some directory which he knows it is rarely change, to not be watched only check it once a day?

The best would be if git annex could automatically adapt itself.
Ie. it watches eg. 200 files, and if some of it does not change for three days, then it drops from the watching basket, and those who changed (noticed while sanity checked) it adds to the basket.

I don't really want to raise the ulimit, because my ultimate goal is to have git-annex on multiple raspberry pi with external harddrive (one at my home, one at my mom's home, one at my friends home, etc, etc). And raspberry is fairly low on resource.

I'm interested in your thoughts.

Best,
 Laszlo

[[!tag /design/assistant]]
[[!meta title="assistant can try to add too many files at once in batch add mode"]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="comment 10"
 date="2013-08-26T18:33:40Z"
 content="""
@Joey: it was a \"pretty large\" transfer, several hundred gigabytes in perhaps ~100000 files.  The copying was going to a GPG-encrypted directory remote.
The error only happened once or twice so far.  Point taken about find in /proc; I'll do that if it happens next time.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="comment 11"
 date="2013-12-16T17:20:34Z"
 content="""
I now understand the problem described in comment 6, where once it started failing, it would leak one file descriptor per failure.

I think that failure mode was fixed by accident in the changes in [[!commit 2fd63f3cfac705f0a18f4bcbe0489ce8ea1800d7]].

This doesn't explain what would open so many files to get it into that failure mode, however.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkipQLNyt8RHREHpg2k5wdYeRSCCvSNSBg"
 nickname="Tim"
 subject="Having "too many open files" message show up"
 date="2014-01-13T14:57:11Z"
 content="""
This is happening to me now about twice per day.  I've got a repo that is completely synced already and I'm working on a project in IntelliJ.  When I check git-annex I often see this message when it is doing a startup scan or consistency check.

The message can be seen here: http://imgur.com/Xb4LA73

Are there logs somewhere that I can gather and supply to you to help track this down?  I'm using the 2014-01-07 release of git-annex for Mac OS.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkipQLNyt8RHREHpg2k5wdYeRSCCvSNSBg"
 nickname="Tim"
 subject="Happened four times yesterday"
 date="2014-01-14T14:33:02Z"
 content="""
Maybe you can e-mail me and we can discuss this?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm5iosFbL2By7UFeViqkc6v-hoAtqILeDA"
 nickname="Laszlo"
 subject="comment 1"
 date="2013-04-18T06:20:13Z"
 content="""
Once resource exhausted git-annex has git [defunct] process too.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 2"
 date="2013-04-23T19:44:04Z"
 content="""
I have tried repeatedly to reproduce this problem, and I cannot.

git-annex does *not* keep every file open. It tends to have less than 10 open file descriptors at any one time.

I thought perhaps `lsof` opened every file, but it does not seem to, either.

So far, I have no indication that the problem had to do with git-annex at all. If some other program on the system opened a great many files, it could cause this to happen to git-annex.

You pasted a debug log that shows that the problem persisted for several minutes. So you should make it happen again, and in that time period, investigate what program has so many files open. You can do this with lsof, or, if lsof won't run, by looking in /proc/$pid/fd/

Or, of course, give me enough information to reproduce the problem. \"I have 23459 files\" isn't much help..
"""]]


code:
[[!comment format=mdwn
 username="http://adamspiers.myopenid.com/"
 nickname="Adam"
 subject="I just reproduced this"
 date="2013-07-26T23:37:26Z"
 content="""
On openSUSE 12.3 with this version (which I'm sure is horribly old):

    git-annex version: 4.20130314
    local repository version: 3
    default repository version: 3
    supported repository versions: 3 4
    upgrade supported from repository versions: 0 1 2
    build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP DNS

I ran 'git annex get' on a large repository, and got this:

    [snipped]
    git-annex: /home/adam/music/.git/annex/transfer/failed/download/9c930baf-6b25-4294-b1f9-a90ba9663fb7/SHA1-s7227476--49d203960b062ec00c0d1d7042c4b6aa6720b976: openFile: resource exhausted (Too many open files)
    failed
    git-annex: runInteractiveProcess: pipe: Too many open files

    git-annex: git: runInteractiveProcess: resource exhausted (Too many open files)
    failed
    git-annex: get: 1207 failed

Then I ran it again and saw that after every file retrieved, git-annex leaks another lockfile.  lsof shows an ever increasing number of files like this:

    [snipped]
    git-annex 32498 adam   86rR  REG  253,3        0   787353 /home/adam/music/.git/annex/ssh/adam@atlantic.lock
    git-annex 32498 adam   87rR  REG  253,3        0   787353 /home/adam/music/.git/annex/ssh/adam@atlantic.lock
    git-annex 32498 adam   88rR  REG  253,3        0   787353 /home/adam/music/.git/annex/ssh/adam@atlantic.lock
    git-annex 32498 adam   89rR  REG  253,3        0   787353 /home/adam/music/.git/annex/ssh/adam@atlantic.lock
    git-annex 32498 adam   90rR  REG  253,3        0   787353 /home/adam/music/.git/annex/ssh/adam@atlantic.lock
    git-annex 32498 adam   91rR  REG  253,3        0   787353 /home/adam/music/.git/annex/ssh/adam@atlantic.lock
    git-annex 32498 adam   92rR  REG  253,3        0   787353 /home/adam/music/.git/annex/ssh/adam@atlantic.lock
    git-annex 32498 adam   93r  FIFO    0,8      0t0 28752703 pipe
    git-annex 32498 adam   94r  FIFO    0,8      0t0 28765910 pipe
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.246.110"
 subject="comment 4"
 date="2013-07-28T00:23:08Z"
 content="""
Hmm, Adam your version is older than the bug reporter's version. OTOH, while there were several FD leak fixes after your version, none of them were to Annex.LockPool, which is what's used for the ssh lock files.

I can't reproduce it with `git annex get` and the current release.. can you?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="I just reproduced this with the latest version"
 date="2013-08-11T03:17:11Z"
 content="""
At some point during a large copy, there's an ever increasing number of pipes in /proc/git-annex-pid/fd
As soon as it hits the limit (1023 in my case), copies start failing

    etc
    r-x------ 1 michael michael 64 Aug 10 20:14 851 -> pipe:[2250609]
    l-wx------ 1 michael michael 64 Aug 10 20:14 852 -> pipe:[2251549]
    lr-x------ 1 michael michael 64 Aug 10 20:14 853 -> pipe:[2251550]
    l-wx------ 1 michael michael 64 Aug 10 20:14 854 -> pipe:[2250612]
    lr-x------ 1 michael michael 64 Aug 10 20:14 855 -> pipe:[2250613]
    l-wx------ 1 michael michael 64 Aug 10 20:14 856 -> pipe:[2246639]
    lr-x------ 1 michael michael 64 Aug 10 20:14 857 -> pipe:[2246640]
    l-wx------ 1 michael michael 64 Aug 10 20:14 858 -> pipe:[2246642]
    lr-x------ 1 michael michael 64 Aug 10 20:14 859 -> pipe:[2246643]
    l-wx------ 1 michael michael 64 Aug 10 20:13 86 -> pipe:[2241378]
    l-wx------ 1 michael michael 64 Aug 10 20:14 860 -> pipe:[2246645]
    lr-x------ 1 michael michael 64 Aug 10 20:14 861 -> pipe:[2246646]
    l-wx------ 1 michael michael 64 Aug 10 20:14 862 -> pipe:[2246648]
    lr-x------ 1 michael michael 64 Aug 10 20:14 863 -> pipe:[2246649]
    l-wx------ 1 michael michael 64 Aug 10 20:14 864 -> pipe:[2246653]
    lr-x------ 1 michael michael 64 Aug 10 20:14 865 -> pipe:[2246654]
    lr-x------ 1 michael michael 64 Aug 10 20:14 866 -> pipe:[2249407]
    l-wx------ 1 michael michael 64 Aug 10 20:14 867 -> pipe:[2251789]
    l-wx------ 1 michael michael 64 Aug 10 20:14 868 -> pipe:[2250627]
    lr-x------ 1 michael michael 64 Aug 10 20:14 869 -> pipe:[2250628]
    lr-x------ 1 michael michael 64 Aug 10 20:13 87 -> pipe:[2241379]
    l-wx------ 1 michael michael 64 Aug 10 20:14 870 -> pipe:[2251778]
    lr-x------ 1 michael michael 64 Aug 10 20:14 871 -> pipe:[2251779]
    l-wx------ 1 michael michael 64 Aug 10 20:14 872 -> pipe:[2251781]
    lr-x------ 1 michael michael 64 Aug 10 20:14 873 -> pipe:[2251782]
    l-wx------ 1 michael michael 64 Aug 10 20:14 874 -> pipe:[2250635]
    lr-x------ 1 michael michael 64 Aug 10 20:14 875 -> pipe:[2250636]
    l-wx------ 1 michael michael 64 Aug 10 20:14 876 -> pipe:[2251575]
    lr-x------ 1 michael michael 64 Aug 10 20:14 877 -> pipe:[2251576]
    l-wx------ 1 michael michael 64 Aug 10 20:14 878 -> pipe:[2251785]
    lr-x------ 1 michael michael 64 Aug 10 20:14 879 -> pipe:[2251786]
    l-wx------ 1 michael michael 64 Aug 10 20:13 88 -> pipe:[2247853]
    l-wx------ 1 michael michael 64 Aug 10 20:14 880 -> pipe:[2249430]
    lr-x------ 1 michael michael 64 Aug 10 20:14 881 -> pipe:[2249431]
    l-wx------ 1 michael michael 64 Aug 10 20:14 882 -> pipe:[2251581]
    lr-x------ 1 michael michael 64 Aug 10 20:14 883 -> pipe:[2251582]
    l-wx------ 1 michael michael 64 Aug 10 20:14 884 -> pipe:[2250653]
    lr-x------ 1 michael michael 64 Aug 10 20:14 885 -> pipe:[2250654]
    lr-x------ 1 michael michael 64 Aug 10 20:14 886 -> pipe:[2251790]
    l-wx------ 1 michael michael 64 Aug 10 20:14 887 -> pipe:[2250670]
    l-wx------ 1 michael michael 64 Aug 10 20:14 888 -> pipe:[2250663]
    lr-x------ 1 michael michael 64 Aug 10 20:14 889 -> pipe:[2250664]
    lr-x------ 1 michael michael 64 Aug 10 20:13 89 -> pipe:[2247854]
    l-wx------ 1 michael michael 64 Aug 10 20:14 890 -> pipe:[2250668]
    lr-x------ 1 michael michael 64 Aug 10 20:14 891 -> pipe:[2250669]
    lr-x------ 1 michael michael 64 Aug 10 20:14 892 -> pipe:[2250671]
    l-wx------ 1 michael michael 64 Aug 10 20:14 894 -> pipe:[2251601]
    lr-x------ 1 michael michael 64 Aug 10 20:14 895 -> pipe:[2251602]
    lr-x------ 1 michael michael 64 Aug 10 20:13 9 -> pipe:[2241268]
    lr-x------ 1 michael michael 64 Aug 10 20:13 90 -> pipe:[2242273]
    l-wx------ 1 michael michael 64 Aug 10 20:13 91 -> pipe:[2245776]
    etc

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="comment 6"
 date="2013-08-11T03:20:37Z"
 content="""
with every

    git annex
    copy blah/blah/blah (to testremote...)
    git-annex: runInteractiveProcess: pipe: Too many open files
    failed

The number of open fd's by git-annex increases by 1.

4.20130802 built with cabal on Ubuntu 13.04

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="comment 7"
 date="2013-08-11T03:21:45Z"
 content="""
...increases in the failing case, that is. This doesn't happen all the time.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="comment 8"
 date="2013-08-11T20:44:19Z"
 content="""
It also looks like the location log has got corrupted (files are actually present, but not recorded in the location log) somewhere along the lines as I was trying to get/drop to figure out what's going on.
Explicitly dropping files then getting files fixes the location log issue.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.63"
 subject="comment 9"
 date="2013-08-24T19:10:21Z"
 content="""
@Michael how large a copy are you doing? And what kind of remote are you copying the files to?
It would be helpful if you could be more specific about something I could do to reproduce the problem. Without a test case, I am unlikely to fix the bug. With a test case, I'd be surprised if it took long to fix it.

If you have a process running that is experiencing the problem, you can also narrow it down a *lot* by looking at what these leaking pipe file descriptors are pipes to.  For example, if you have:

lr-x------ 1 michael michael 64 Aug 10 20:14 895 -> pipe:[2251602]

You can run `find /proc/ -ls 2251602` and find the process at other end of the pipe, and look its pid up in ps to see what command it is.
"""]]


code:
### Please describe the problem.

Somewhere on the forums you advised that in order to revert to an old
version of a file in a direct-mode annex, one should briefly switch to
indirect mode and check out an earlier revision.

This won’t work though if the direct-mode repo is on a VFAT
file-system, which does not support indirect mode.

If I didn’t overlook an alternative way, would it make sense to expose
this functionality in «git annex get $file»? Like «git annex get $file
--from $repo/$revision»?

Specifying revisions to get from might also be convenient for
alternative use-cases, I suppose.

Best regards,
T.

### What version of git-annex are you using? On what operating system?

git-annex 5.20140421
Linux 3.14.3

> VFAT is a crippled file system. I am not going to reimplenent all of git
> to support it. The suggested method won't work; git-annex would need
> to add a git-annex revert, etc, etc, etc ad infinitum. [[wontfix|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="zardoz"
 ip="92.227.51.179"
 subject="comment 1"
 date="2014-05-13T20:43:38Z"
 content="""
Crosslink to related wishlist item.
http://git-annex.branchable.com/todo/A_Way_To_Extract_Previous_Versions_of_a_File_From_a_Direct_Repo/
"""]]


code:
[[!comment format=mdwn
 username="zardoz"
 ip="92.227.51.179"
 subject="comment 2"
 date="2014-05-17T14:16:12Z"
 content="""
One follow-up comment, because it occurred to me I might have phrased
the bug inadequately. By «revert» I didn’t mean a revert in the
git-sense, but merely «checking out» a file from a selected commit.

But I think I can achieve this in direct mode on VFAT by doing «git
show upstream/branch~X:file > file», right? «git annex get file» then
gets the correct content, only that it will not place the content into
the indicated file until I do «git annex fsck». Maybe «git annex get»
could take this into account and have the content immediately stored
there.

"""]]


code:
[[!comment format=mdwn
 username="zardoz"
 ip="92.227.51.179"
 subject="comment 3"
 date="2014-05-17T14:21:30Z"
 content="""
Plus it seems that the file can’t be dropped normally afterwards.
"""]]


code:
### Please describe the problem.
The remote server, connected with rsync and with encryption enabled doesn't fill with files.

### What steps will reproduce the problem?
* Add remote server via webapp
* Supply password when asked
* both buttons turn green ('ready to add remote server')
* Select encrypted rsync repository
* When done, files will be queued for transfer, and the queue empties quickly. Afterwards, no files have actually been transferred, but a green message appears and says something like 'synced with xxx'.
* Also, on the remote an empty directory (~/annex) is created.

### What version of git-annex are you using? On what operating system?
local: kubuntu 12.10,            git-annex 4.20130621-g36258de
remote: debian (linux 3.8.0-25), git-annex 4.20130621-g36258de
Both are installed from tarball and PATH is set at the top of .bashrc.

### Please provide any additional information below.

[[!format sh """

Here is what is put in the logs when the button is toggled from 'syncing disabled' to 'syncing enabled'.
daemon.log:

[2013-07-03 13:43:07 CEST] call: git ["--git-dir=/home/boris/annex/.git","--work-tree=/home/boris/annex","config","remote.mybox.annex-sync","true"]
[2013-07-03 13:43:07 CEST] read: git ["config","--null","--list"]
[2013-07-03 13:43:07 CEST] read: git ["config","--null","--list"]
[2013-07-03 13:43:07 CEST] main: Syncing with mybox
[2013-07-03 13:43:07 CEST] read: git ["--git-dir=/home/boris/annex/.git","--work-tree=/home/boris/annex","symbolic-ref","HEAD"]
[2013-07-03 13:43:07 CEST] read: git ["--git-dir=/home/boris/annex/.git","--work-tree=/home/boris/annex","show-ref","refs/heads/master"]
[2013-07-03 13:43:07 CEST] read: git ["--git-dir=/home/boris/annex/.git","--work-tree=/home/boris/annex","show-ref","git-annex"]
[2013-07-03 13:43:07 CEST] read: git ["--git-dir=/home/boris/annex/.git","--work-tree=/home/boris/annex","show-ref","--hash","refs/heads/git-annex"]
[2013-07-03 13:43:07 CEST] read: git ["--git-dir=/home/boris/annex/.git","--work-tree=/home/boris/annex","log","refs/heads/git-annex..4cc51b410f5257f60e4ea187ab0c29783effcc88","--oneline","-n1"]
[2013-07-03 13:43:07 CEST] read: git ["--git-dir=/home/boris/annex/.git","--work-tree=/home/boris/annex","log","refs/heads/git-annex..6728d4d49ef97365eea0e2d379951acee9a9ded8","--oneline","-n1"]
[2013-07-03 13:43:07 CEST] read: git ["--git-dir=/home/boris/annex/.git","--work-tree=/home/boris/annex","symbolic-ref","HEAD"]
[2013-07-03 13:43:07 CEST] read: git ["--git-dir=/home/boris/annex/.git","--work-tree=/home/boris/annex","show-ref","refs/heads/master"]
[2013-07-03 13:43:07 CEST] TransferScanner: starting scan of [Remote { name ="mybox" }]
[2013-07-03 13:43:07 CEST] read: git ["--git-dir=/home/boris/annex/.git","--work-tree=/home/boris/annex","ls-files","--cached","-z","--"]
[2013-07-03 13:43:07 CEST] TransferScanner: queued Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/TSPC_FF_R.png Nothing : expensive scan found missing object
[2013-07-03 13:43:07 CEST] Transferrer: Transferring: Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/TSPC_FF_R.png Nothing
[2013-07-03 13:43:07 CEST] TransferScanner: queued Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/Thumbs.db Nothing : expensive scan found missing object
[2013-07-03 13:43:07 CEST] TransferWatcher: transfer starting: Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/TSPC_FF_R.png Nothing
[2013-07-03 13:43:07 CEST] TransferScanner: queued Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/administratie/Aanmeldingsformulier Masterexamen.odt Nothing : expensive scan found missing object
[2013-07-03 13:43:07 CEST] TransferScanner: queued Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/administratie/Intake_form_MSc_Electrical_Engineering_def dec 2011-1.pdf Nothing : expensive scan found missing object
[2013-07-03 13:43:07 CEST] TransferScanner: queued Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/administratie/Opdrachtomschrijving Master.doc Nothing : expensive scan found missing object
[2013-07-03 13:43:07 CEST] TransferScanner: queued Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/administratie/Opdrachtomschrijving Master.pdf Nothing : expensive scan found missing object
[2013-07-03 13:43:07 CEST] TransferScanner: queued Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/administratie/uren.xls Nothing : expensive scan found missing object
[2013-07-03 13:43:07 CEST] TransferScanner: queued Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/eldo_ur.pdf Nothing : expensive scan found missing object
[2013-07-03 13:43:07 CEST] TransferScanner: queued Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/literatuur/ A 65J-Conversion-Step 0-to-50MS 0-to-0.7mW 9b Charge-Sharing SAR ADC in 90nm Digital CMOS.pdf Nothing : expensive scan found missing object
[2013-07-03 13:43:07 CEST] TransferScanner: queued Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/literatuur/.directory Nothing : expensive scan found missing object
[2013-07-03 13:43:07 CEST] TransferScanner: queued Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/literatuur/2009SOVC_A_0.92mW_10-bit_50-MSs_SAR_ADC_in_0.13um_CMOS_Process.pdf Nothing : expensive scan found missing object
[2013-07-03 13:43:07 CEST] TransferScanner: queued Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/literatuur/A 8-bit 500-KSs Low Power SAR ADC for Biomedical Applications.pdf Nothing : expensive scan found missing object
fatal: unrecognized command 'rsync --server -vre.iLsf --partial-dir .rsync-partial . annex/'
git-annex-shell: git-shell failed
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]
[2013-07-03 13:43:07 CEST] TransferWatcher: transfer finishing: Transfer {transferDirection = Upload, transferUUID = UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23", transferKey = Key {keyName = "ea86501fb0db033a103ed2c0806a1bddc145224afc4eb5e17fceb70bf1f674da.png", keyBackendName = "SHA256E", keySize = Just 11004, keyMtime = Nothing}}
fatal: unrecognized command 'rsync --server -vre.iLsf --partial-dir .rsync-partial . annex/'
git-annex-shell: git-shell failed
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]
[2013-07-03 13:43:07 CEST] TransferScanner: queued Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/literatuur/A 9.2b 47fJ SAR with input range prediction DAC switching.pdf Nothing : expensive scan found missing object
[2013-07-03 13:43:07 CEST] Transferrer: Transferring: Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/Thumbs.db Nothing
[2013-07-03 13:43:07 CEST] TransferWatcher: transfer starting: Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/Thumbs.db Nothing
fatal: unrecognized command 'rsync --server -vre.iLsf --partial-dir .rsync-partial . annex/'
git-annex-shell: git-shell failed
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]
[2013-07-03 13:43:07 CEST] TransferWatcher: transfer finishing: Transfer {transferDirection = Upload, transferUUID = UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23", transferKey = Key {keyName = "23b9a2be728c8af402ededb3eef7e9238b38d54c0ca50a05cbdf4aeea8f03c76.db", keyBackendName = "SHA256E", keySize = Just 12800, keyMtime = Nothing}}
fatal: unrecognized command 'rsync --server -vre.iLsf --partial-dir .rsync-partial . annex/'
git-annex-shell: git-shell failed
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]
:
[2013-07-03 13:43:07 CEST] TransferScanner: queued Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/literatuur/A Low-Power Static Dual Edge-Triggered Flip-Flop.pdf Nothing : expensive scan found missing object
[2013-07-03 13:43:07 CEST] Transferrer: Transferring: Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/administratie/Aanmeldingsformulier Masterexamen.odt Nothing
[2013-07-03 13:43:07 CEST] TransferWatcher: transfer starting: Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/administratie/Aanmeldingsformulier Masterexamen.odt Nothing
fatal: unrecognized command 'rsync --server -vre.iLsf --partial-dir .rsync-partial . annex/'
git-annex-shell: git-shell failed
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]
[2013-07-03 13:43:07 CEST] TransferWatcher: transfer finishing: Transfer {transferDirection = Upload, transferUUID = UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23", transferKey = Key {keyName = "f27ac57481c9eb0bf7bf1f6c99666aca5c8282137effa33791241b3d36b12de2.odt", keyBackendName = "SHA256E", keySize = Just 22041, keyMtime = Nothing}}
fatal: unrecognized command 'rsync --server -vre.iLsf --partial-dir .rsync-partial . annex/'
git-annex-shell: git-shell failed
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]
[2013-07-03 13:43:07 CEST] TransferScanner: queued Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/literatuur/An Energy-Efficient Charge Recycling Approach for a SAR Converter With Capacitive DAC.pdf Nothing : expensive scan found missing object
[2013-07-03 13:43:07 CEST] Transferrer: Transferring: Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/administratie/Intake_form_MSc_Electrical_Engineering_def dec 2011-1.pdf Nothing
[2013-07-03 13:43:07 CEST] TransferWatcher: transfer starting: Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/administratie/Intake_form_MSc_Electrical_Engineering_def dec 2011-1.pdf Nothing
fatal: unrecognized command 'rsync --server -vre.iLsf --partial-dir .rsync-partial . annex/'
git-annex-shell: git-shell failed
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]
[2013-07-03 13:43:07 CEST] TransferWatcher: transfer finishing: Transfer {transferDirection = Upload, transferUUID = UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23", transferKey = Key {keyName = "a261cb3835ed869c6ad2347d303ed10e94efa8a50cc9ff053d772c2158097244.pdf", keyBackendName = "SHA256E", keySize = Just 667666, keyMtime = Nothing}}
fatal: unrecognized command 'rsync --server -vre.iLsf --partial-dir .rsync-partial . annex/'
git-annex-shell: git-shell failed
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]
[2013-07-03 13:43:07 CEST] TransferScanner: queued Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/literatuur/Calibration Technique for SAR Analog-to-Digital Converters.pdf Nothing : expensive scan found missing object
[2013-07-03 13:43:07 CEST] Transferrer: Transferring: Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/administratie/Opdrachtomschrijving Master.doc Nothing
[2013-07-03 13:43:07 CEST] TransferWatcher: transfer starting: Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/administratie/Opdrachtomschrijving Master.doc Nothing
fatal: unrecognized command 'rsync --server -vre.iLsf --partial-dir .rsync-partial . annex/'
git-annex-shell: git-shell failed
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]
[2013-07-03 13:43:08 CEST] TransferWatcher: transfer finishing: Transfer {transferDirection = Upload, transferUUID = UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23", transferKey = Key {keyName = "cb69f2f4ae313277acd163b9bba19392ce50d3c5205ba8bbd90bf9c453088176.doc", keyBackendName = "SHA256E", keySize = Just 34304, keyMtime = Nothing}}
[2013-07-03 13:43:08 CEST] call: git ["--git-dir=/home/boris/annex/.git","--work-tree=/home/boris/annex","config","remote.mybox.annex-sync","false"]
[2013-07-03 13:43:08 CEST] read: git ["config","--null","--list"]
[2013-07-03 13:43:08 CEST] read: git ["config","--null","--list"]
fatal: unrecognized command 'rsync --server -vre.iLsf --partial-dir .rsync-partial . annex/'
git-annex-shell: git-shell failed
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]
[2013-07-03 13:43:08 CEST] TransferScanner: queued Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/literatuur/D15_01.pdf Nothing : expensive scan found missing object
[2013-07-03 13:43:08 CEST] Transferrer: Transferring: Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/administratie/Opdrachtomschrijving Master.pdf Nothing
[2013-07-03 13:43:08 CEST] TransferWatcher: transfer starting: Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/administratie/Opdrachtomschrijving Master.pdf Nothing
fatal: unrecognized command 'rsync --server -vre.iLsf --partial-dir .rsync-partial . annex/'
git-annex-shell: git-shell failed
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]
[2013-07-03 13:43:08 CEST] TransferWatcher: transfer finishing: Transfer {transferDirection = Upload, transferUUID = UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23", transferKey = Key {keyName = "cea054d80cb5ca572b7055c0132ec2d13125534566ed4a79ab2514c9a60d8ee4.pdf", keyBackendName = "SHA256E", keySize = Just 19802, keyMtime = Nothing}}
fatal: unrecognized command 'rsync --server -vre.iLsf --partial-dir .rsync-partial . annex/'
git-annex-shell: git-shell failed
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]
[2013-07-03 13:43:08 CEST] Transferrer: Transferring: Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/literatuur/D15_01.pdf Nothing
[2013-07-03 13:43:08 CEST] TransferWatcher: transfer starting: Upload UUID "32841e5f-1e4d-4c72-84c4-bbb54a335a23" afstuderen/literatuur/D15_01.pdf Nothing
fatal: unrecognized command 'rsync --server -vre.iLsf --partial-dir .rsync-partial . annex/'
git-annex-shell: git-shell failed
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]

# End of transcript or log.
"""]]

> [[fixed|done]], corrected logic error that caused `authorized_keys`
> to incorrectly force the git-annex-shell command for rsync remotes. --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.254.222"
 subject="comment 1"
 date="2013-07-08T16:55:09Z"
 content="""
Finally got a chance to look into this.

The ~/.ssh/authorized_keys file on the server has been set up wrong by the webapp. If you look at it, it'll contain a part like: command=\"GIT_ANNEX_SHELL_DIRECTORY=... ~/.ssh/git-annex-shell\"

That can't be used for rsync. If you delete that command= part, your remote should start working.
"""]]


code:
### Please describe the problem.

As described in [[tips/Internet_Archive_via_S3]], there is a problem using S3 with buckets that have capital letters. The bug lies either in the hS3 library or in archive.org itself.

### What steps will reproduce the problem?

Try to add an [[special_remotes/S3]] remote with capital letters in the bucket name.

### What version of git-annex are you using? On what operating system?

[[!format txt """
git-annex version: 4.20130921-g434dc22
build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP DNS Feeds Quvi
local repository version: 3
default repository version: 3
supported repository versions: 3 4
upgrade supported from repository versions: 0 1 2
"""]]

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
anarcat@angela:video$ git annex initremote archive-moglenrepublica type=S3 host=s3.us.archive.org bucket=Republica2012-EbenMoglen-FreedomOfThoughtRequiresFreeMedia
initremote archive-moglenrepublica (Internet Archive mode) git-annex: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. For more information, see REST Authentication and SOAP Authentication for details.
# End of transcript or log.
"""]]

Just thought it would be better to have a separate thread for this bug. :)

> [[fixed|done]] --[[Joey]]


code:
S3 has memory leaks

Sending a file to S3 causes a slow memory increase toward the file size.

Copying the file back from S3 causes a slow memory increase toward the
file size.

The author of hS3 is aware of the problem, and working on it. I think I
have identified the root cause of the buffering; it's done by hS3 so it can
resend the data if S3 sends it a 307 redirect. --[[Joey]]

At least the send leak should be fixed by the patch in the s3-memory-leak
branch in git. That needs a patch to hS3, which I have sent to its author.
--[[Joey]]
[[!tag confirmed]]


code:
[[!comment format=mdwn
 username="http://schnouki.net/"
 nickname="Schnouki"
 subject="comment 1"
 date="2013-10-18T08:36:45Z"
 content="""
Hi Joey,

It looks like your patch hasn't been merged yet. And this bug is quite annoying for me (can't backup files bigger than 1.5 GB from my NAS).

Would it be possible to include this fix in your standalone builds and Debian packages?

Thanks!
"""]]


code:
[[!comment format=mdwn
 username="http://schnouki.net/"
 nickname="Schnouki"
 subject="comment 2"
 date="2014-01-29T01:19:27Z"
 content="""
Any news about this?
"""]]


code:
What steps will reproduce the problem?

> Try to copy/move a file greater than 5G to S3.

    git annex copy large_file.tgz --to cloud

What is the expected output? What do you see instead?

> Looks like git-annex may not be using the Multipart Upload API: http://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html

> Expected transfer to succeed, instead this error is output:

    copy large-file.tgz (gpg) (checking cloud...) (to cloud...) Reading passphrase from file descriptor 12

     Your proposed upload exceeds the maximum allowed size
     failed
     git-annex: copy: 1 failed

What version of git-annex are you using? On what operating system?

> OSX 10.8.2

Please provide any additional information below.

    annex [master●] % git annex status
	supported backends: SHA256E SHA1E SHA512E SHA224E SHA384E SHA256 SHA1 SHA512 SHA224 SHA384 WORM URL
	supported remote types: git S3 bup directory rsync web webdav glacier hook
	repository mode: indirect
	trusted repositories: 0
	semitrusted repositories: 3
		00000000-0000-0000-0000-000000000001 -- web
	 	BE1D8EC7-C64B-47DE-AD4E-2A50437532B4 -- cloud
	 	E84568BA-6A4B-4AA1-B622-605B9248EDB1 -- here (eric laptop)
	untrusted repositories: 0
	dead repositories: 0
	transfers in progress: none
	available local disk space: 169 gigabytes (+1 megabyte reserved)
	temporary directory size: 218 megabytes (clean up with git-annex unused)
	local annex keys: 24
	local annex size: 8 gigabytes
	known annex keys: 25
	known annex size: 8 gigabytes
	bloom filter size: 16 mebibytes (0% full)
	backend usage:
		SHA256E: 49
	annex [master●] % git annex version
	git-annex version: 3.20130114
	local repository version: 3
	default repository version: 3
	supported repository versions: 3
	upgrade supported from repository versions: 0 1 2

[[!tag confirmed]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 1"
 date="2014-05-13T15:09:42Z"
 content="""
JFTR, this is impacting DebConf's video storage as well.

Richard
"""]]


code:
[[!comment format=mdwn
 username="annexuser"
 ip="64.71.7.82"
 subject="comment 2"
 date="2014-07-14T18:21:00Z"
 content="""
I'm having the same problem. Is there a fix for this yet?

    $ git annex version
    git-annex version: 5.20140709-gc75193e
    build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV Inotify DBus DesktopNotify XMPP DNS Feeds Quvi TDFA CryptoHash
    key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
    remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier ddar hook external
    local repository version: 5
    supported repository version: 5
    upgrade supported from repository versions: 0 1 2 4
"""]]


code:
### Please describe the problem.
On startup, the webapp shows the following warning in a box to the upper right of the screen, with an offer to "Restart thread":

    SanityCheckerStartup crashed: /home/anton/Halvhemligt/.git/annex/misctmp/IMG_32978856.JPG: getFileStatus: does not exist (No such file or directory)

Restarting it causes it to crash again immediately. The log shows the same:

    SanityCheckerStartup crashed: /home/anton/Halvhemligt/.git/annex/misctmp/IMG_32978856.JPG: getFileStatus: does not exist (No such file or directory)
    [2014-06-02 23:31:13 CEST] SanityCheckerStartup: warning SanityCheckerStartup crashed: /home/anton/Halvhemligt/.git/annex/misctmp/IMG_32978856.JPG: getFileStatus: does not exist (No such file or directory)

/home/anton/Halvhemligt/.git/annex/misctmp/ contains the following:

    lrwxrwxrwx 2 anton anton 199 15 maj 21.08 IMG_32978856.JPG -> ../.git/annex/objects/z3/K2/SHA256E-s728022--12de1f194042af3f8c4dbee15c317de0511bbb8b9e8a0463fffb07e7bbc58bb5.JPG/SHA256E-s728022--12de1f194042af3f8c4dbee15c317de0511bbb8b9e8a0463fffb07e7bbc58bb5.JPG
    lrwxrwxrwx 2 anton anton 199 15 maj 21.09 IMG_32988856.JPG -> ../.git/annex/objects/vw/30/SHA256E-s688301--8bb6d636163b443705c9a333194116da3937d8272b70613ca6345eaf6bba1255.JPG/SHA256E-s688301--8bb6d636163b443705c9a333194116da3937d8272b70613ca6345eaf6bba1255.JPG
    lrwxrwxrwx 2 anton anton 199 15 maj 21.17 IMG_33198856.JPG -> ../.git/annex/objects/p3/WG/SHA256E-s754900--224e6489370527156293912e11390af517ad4ef9374ee22c8324b5af5fac0dd7.JPG/SHA256E-s754900--224e6489370527156293912e11390af517ad4ef9374ee22c8324b5af5fac0dd7.JPG

The symlinks are all broken. IMG_3297.JPG, IMG_3298.JPG and IMG_3319.JPG exist in the repository, but I do not recognize the appended numbers 8856. git log shows that these three files have not been modified since they were first added.

### What steps will reproduce the problem?
No idea. I believe this repository was created in the webapp but most of the changes to it has been done in the CLI.

### What version of git-annex are you using? On what operating system?
Precompiled 5.20140530 on Arch Linux x86_64 (git-annex-bin package from the AUR), but 5.20140518 had the same problem.

### Please provide any additional information below.
The crash has not caused any real problem for me that I'm aware of. I'm just reporting it because the error message looks a little scary.

> [[dup|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 1"
 date="2014-06-04T18:07:11Z"
 content="""
This was already fixed in [[!commit 9c0f3ae7e13cc8ed43ac73fbab565e07c10d2c90]].
"""]]


code:
[[!comment format=mdwn
 username="https://launchpad.net/~eliasson"
 nickname="eliasson"
 subject="comment 2"
 date="2014-06-05T17:15:32Z"
 content="""
Great. I will patiently await the next release.
"""]]


code:
### Please describe the problem.
Entering a jabber address which's server got a selfsigned certificate, the process just fails, without asking for acceptance for that certificate. This is quite a showstopper.
(for example: jabber.ccc.de)

### What steps will reproduce the problem?
Try with an account from e.g. jabber.ccc.de

### What version of git-annex are you using? On what operating system?
Arch Linux, aur/git-annex-standalone 4.20130709-1

### Please provide any additional information below.
There is no logoutput to add... I'm sorry.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

# End of transcript or log.
"""]]

[[!meta title="XMPP does not work with jabber.ccc.de"]]

[[!tag moreinfo]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.90"
 subject="comment 1"
 date="2013-07-16T17:33:23Z"
 content="""
Hmm, actually the XMPP library it's using does not fail on an invalid or self-signed cert. This is something that SSL using libraries sadly often get wrong, and yeah, I see I contacted the library author a while ago, and a \"sessionIsSecure\" has been added to it, which can be used to check if the cert was valid.

This leaves open the question of what's happening with you on the CCC server. Did you get an error message?

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.90"
 subject="comment 2"
 date="2013-07-16T17:42:42Z"
 content="""
Actually, sessionIsSecure only tells me if it's using SSL at all; still waiting on the XMPP library to add cert checking..
"""]]


code:
[[!comment format=mdwn
 username="https://launchpad.net/~psycojoker"
 nickname="psycojoker"
 subject="comment 3"
 date="2013-11-25T09:54:33Z"
 content="""
Just to confirm that I'm also affected. Maybe you should add the possibility in the error message next to \"is the password correct?\".
"""]]


code:
[[!comment format=mdwn
 username="http://xlogon.net/yminus"
 nickname="http://xlogon.net/yminus"
 subject="comment 4"
 date="2014-05-08T21:12:49Z"
 content="""
When will this issue be fixed? Currently my Jabber account is on jabber.ccc.de.

This is the error message:

    Unable to connect to the Jabber server. Maybe you entered the wrong password? (Error message: host jabberd.jabber.ccc.de.:5222 failed: AuthenticationFailure (Element {elementName = Name {nameLocalName = \"failure\", nameNamespace = Just \"urn:ietf:params:xml:ns:xmpp-sasl\", namePrefix = Nothing}, elementAttributes = [], elementNodes = [NodeElement (Element {elementName = Name {nameLocalName = \"bad-protocol\", nameNamespace = Just \"urn:ietf:params:xml:ns:xmpp-sasl\", namePrefix = Nothing}, elementAttributes = [], elementNodes = []})]}); host jabberd.jabber.ccc.de.:80 failed: AuthenticationFailure (Element {elementName = Name {nameLocalName = \"failure\", nameNamespace = Just \"urn:ietf:params:xml:ns:xmpp-sasl\", namePrefix = Nothing}, elementAttributes = [], elementNodes = [NodeElement (Element {elementName = Name {nameLocalName = \"bad-protocol\", nameNamespace = Just \"urn:ietf:params:xml:ns:xmpp-sasl\", namePrefix = Nothing}, elementAttributes = [], elementNodes = []})]}))

Can you advise me another trustworthy non-profit and free (free as in free beer\" as well as \"free as in free speech\") jabber server?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.36"
 subject="comment 5"
 date="2014-05-24T20:03:46Z"
 content="""
AFAIKS, this is not a failure from a self-signed certificate. git-annex contacted the jabber server, accepted its certificate, and tried to log in. The jabber server rejected the login.

This kind of problem has in the past turned out to involve incompatabilities in SASL authentication using some newish authentication schemes. Often it's been a bug in the XMPP server that's tickled by the haksell TLS library. See for example <http://bugs.debian.org/717463>

So, what XMPP server is used by CCC?
"""]]