code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.43"
 subject="comment 1"
 date="2014-01-13T18:57:48Z"
 content="""
The log file is supposed to be stored by git-annex in the git-annex branch of the git repository. I don't know how it could end up being committed to the master branch instead.

Can you run these same commands again and reproduce the problem?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnsuhFUIfWNT-Q-C02FDaSQqceFDge5M9w"
 nickname="Florian"
 subject="comment 2"
 date="2014-01-14T12:12:52Z"
 content="""
Yes, I can always reproduce it with running \"sh test_ga.sh\" with test_ga.sh being:

    set -x

    mkdir GA1
    cd GA1
    git init
    git annex init \"GA1\"
    echo \"GA1\" > testfile
    git annex add .
    git annex sync
    cd ..

    git clone GA1 GA2
    cd GA2
    git annex init \"GA2\"
    git annex get .
    cat testfile
    git annex unlock testfile
    echo \"GA2\" > testfile
    git annex sync
    cd ../GA1

    git annex sync
    cd ..

which gives:

    florian@horus ~/test % ll GA1
    insgesamt 4
    drwxr-x--- 1 florian florian   6 14. Jan 13:07 e6a/
    lrwxrwxrwx 1 florian florian 178 14. Jan 13:07 testfile -> .git/annex/objects/V6/ZM/SHA256E-s4--241dca52164d30bee7a0be8c097f02e382acff765a9d4136718a37abd437a57c/SHA256E-s4--241dca52164d30bee7a0be8c097f02e382acff765a9d4136718a37abd437a57c

with testfile being a dead link. Complete output of the script is at http://pastebin.com/VktWAp2V

I would be glad to help furthermore.
"""]]

repo_name: kakafarm/git

license: unknown


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.43"
 subject="comment 3"
 date="2014-01-14T18:55:11Z"
 content="""
This script does not produce the problem for me. How did you install git-annex from the tarball?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.43"
 subject="comment 4"
 date="2014-01-14T19:00:33Z"
 content="""
Actually, I was able to reproduce this using the standalone tarball and runshell.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.43"
 subject="comment 5"
 date="2014-01-14T19:14:35Z"
 content="""
Analysis: pre-commit hook is run with `GIT_INDEX_FILE=.git/index.lock`; this seems to be causing git-annex branch commits to land in that index file instead of where they are supposed to go. Also, commits of the files in the work tree also end up adding the symlinks to the git-annex branch when this happens. Apparantly because again it's using the wrong index file when committing to the git-annex branch.

So, an easy workaround (disable pre-commit hook or make it `unset GIT_INDEX_FILE`, but I need to dig into why this env var is messing with git-annex like this.. I thought git-annex set the env var itself!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.43"
 subject="comment 6"
 date="2014-01-14T19:16:15Z"
 content="""
Problem is that git-annex prepends `GIT_INDEX_FILE` to the environment; it does not overwrite it. So environment ends up with 2 copies of the var, and which will be used is undefined.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.43"
 subject="comment 7"
 date="2014-01-14T19:33:28Z"
 content="""
I have fixed this in git. I will be moving this thread from the forum to the bug report tracker.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnsuhFUIfWNT-Q-C02FDaSQqceFDge5M9w"
 nickname="Florian"
 subject="comment 8"
 date="2014-01-14T21:15:51Z"
 content="""
Wow, thanks a lot! The fix will be included in the next release of the tarballs?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.43"
 subject="comment 9"
 date="2014-01-14T21:17:16Z"
 content="""
It will be included in the next build of the daily built tarballs. Within an hour of now.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.2.134"
 subject="comment 1"
 date="2013-09-13T19:18:26Z"
 content="""
The Windows port has improved quite a lot since this bug was filed. Can you still reproduce it?

If you install the pre-built git-annex for Windows, you can run `git annex test` to run the test suite.  Does that work? It works for some others on Windows.
"""]]


code:
### Please describe the problem.

Using the latest build, when the agent is started, it works ... and works ... and works ... until it silently dies at some point.

### What steps will reproduce the problem?

I hope this will reproduce:

* Create a remote with an invalid host which is not reachable
* Start git-annex-autostart.vbs
* Wait 5-30 minutes until no git.exe/git-annex.exe are visible any more in task manager (for my current test it took 10 minutes)
* Inspect the logs

### What version of git-annex are you using? On what operating system?

    $ git annex version
    git-annex version: 5.20140715-g622a376
    build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV DNS Feeds Quvi TDFA CryptoHash
    key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
    remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier ddar hook external
    local repository version: 5
    supported repository version: 5
    upgrade supported from repository versions: 2 3 4

### Please provide any additional information below.

I hope/think that this is also related to what I wrote in:
https://git-annex.branchable.com/forum/Restricting_SSH_+_supply_key

Here the assistant apparently takes the wrong ssh.exe (from bin/ instead than cmd/ directory). This results in the fact that for some reason .ssh/config is not used.
In any case, the host alias can not be resolved.
At some point git annex assistant seems to die because of that, but I am not sure.
In any case, a non-reachable server should not be the cause that the assistant dies.
The last entries of daemon.log are always:

    git-annex: <stdin>: hGetLine: invalid argument (Bad file descriptor)
    failed
    git-annex: assistant: 1 failed

So I think this is the cause.

[[!format sh """
[2014-07-14 21:21:19 Pacific Daylight Time] main: starting assistant version 5.20140715-g622a376
[2014-07-14 21:21:19 Pacific Daylight Time] Cronner: You should enable consistency checking to protect your data.
[2014-07-14 21:21:24 Pacific Daylight Time] TransferScanner: Syncing with 192.168.200.121
(scanning...) [2014-07-14 21:21:25 Pacific Daylight Time] Watcher: Performing startup scan
ssh: Could not resolve hostname annex: hostname nor servname provided, or not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
ssh: Could not resolve hostname annex: hostname nor servname provided, or not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
sssh: Could not resolve hostname annex: hostname nor servname provided, or not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
ssh: Could not resolve hostname annex: hostname nor servname provided, or not known

ssh: Could not resolve hostname annex: hostname nor servname provided, or not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
sssh: Could not resolve hostname annex: hostname nor servname provided, or not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
(started...) [2014-07-14 21:21:43 Pacific Daylight Time] Committer: Committing changes to git
[2014-07-14 21:21:43 Pacific Daylight Time] Pusher: Syncing with 192.168.200.121
ssh: Could not resolve hostname annex: hostname nor servname provided, or not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
ssh: Could not resolve hostname annex: hostname nor servname provided, or not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
ssh: Could notssh: Could not resolve hostname annex: hostname nor servname provided, or not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
[2014-07-14 21:22:03 Pacific Daylight Time] Committer: Committing changes to git
fatal: Unable to create 'c:\data\annex\.git/index.lock': File exists.

If no other git process is currently running, this probably means a
git process crashed in this repository earlier. Make sure no other git
process is running and remove the file manually to continue.
fatal: Unable to create 'c:\data\annex\.git/index.lock': File exists.

If no other git process is currently running, this probably means a
git process crashed in this repository earlier. Make sure no other git
process is running and remove the file manually to continue.
Committer crashed: failed to read sha from git write-tree
[2014-07-14 21:22:03 Pacific Daylight Time] Committer: warning Committer crashed: failed to read sha from git write-tree
ssh: Could not resolve hostname annex: hostname nor servname provided, or not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
ssh: Could not resolve hostname annex: hostname nor servname provided, or not known

ssh: Could not resolve hostname annex: hostname nor servname provided, or not known

ssh: Could not resolve hostname annex: hostname nor servname provided, or not known

ssh: Could not resolve hostname annex: hostname nor servname provided, or not known

ssh: Could not resolve hostname annex: hostname nor servname provided, or not known

Network.Socket.ByteString.send: does not exist (No such file or directory)
Network.Socket.ByteString.send: does not exist (No such file or directory)

(Recording state in git...)
(Recording state in git...)

git-annex: <stdin>: hGetLine: invalid argument (Bad file descriptor)
failed
git-annex: assistant: 1 failed
"""]]

[[!tag confirmed]]

> [[fixed|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 1"
 date="2014-07-15T17:36:22Z"
 content="""
I doubt that it has anything to do with a ssh remote.

Please enable debug logging and see if you can get a more detailed log around when it dies.
"""]]


code:
[[!comment format=mdwn
 username="divB"
 ip="128.12.90.218"
 subject="comment 2"
 date="2014-07-15T20:51:07Z"
 content="""
Unfortunately not. I did that already. At least in my opinion nothing special.

These are the latest lines:

[[!format sh \"\"\"
Please make sure you have the correct access rights
and the repository exists.
[2014-07-15 13:34:32 Pacific Daylight Time] read: git [\"--git-dir=c:\\data\\annex\\.git\",\"--work-tree=c:\\data\\annex\",\"-c\",\"core.bare=false\",\"show-ref\",\"git-annex\"]
[2014-07-15 13:34:32 Pacific Daylight Time] read: git [\"--git-dir=c:\\data\\annex\\.git\",\"--work-tree=c:\\data\\annex\",\"-c\",\"core.bare=false\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
[2014-07-15 13:34:32 Pacific Daylight Time] read: git [\"--git-dir=c:\\data\\annex\\.git\",\"--work-tree=c:\\data\\annex\",\"-c\",\"core.bare=false\",\"log\",\"refs/heads/git-annex..8ea0bbec209e1ec1240aa94ba2d6975023d83ac5\",\"-n1\",\"--pretty=%H\"]
[2014-07-15 13:34:32 Pacific Daylight Time] read: git [\"--git-dir=c:\\data\\annex\\.git\",\"--work-tree=c:\\data\\annex\",\"-c\",\"core.bare=false\",\"log\",\"refs/heads/git-annex..528b6b61318d138db5df2262ee501d6895b859d0\",\"-n1\",\"--pretty=%H\"]
[2014-07-15 13:34:32 Pacific Daylight Time] read: git [\"--git-dir=c:\\data\\annex\\.git\",\"--work-tree=c:\\data\\annex\",\"-c\",\"core.bare=false\",\"log\",\"refs/heads/git-annex..45aff729d5c9e60d925fad4c7fd75497ff5301fa\",\"-n1\",\"--pretty=%H\"]
[2014-07-15 13:34:32 Pacific Daylight Time] call: git [\"--git-dir=c:\\data\\annex\\.git\",\"--work-tree=c:\\data\\annex\",\"-c\",\"core.bare=false\",\"show-ref\",\"--verify\",\"-q\",\"refs/remotes/server/annex/direct/master\"]
[2014-07-15 13:34:32 Pacific Daylight Time] read: git [\"--git-dir=c:\\data\\annex\\.git\",\"--work-tree=c:\\data\\annex\",\"-c\",\"core.bare=false\",\"log\",\"refs/heads/annex/direct/master..refs/remotes/server/annex/direct/master\",\"-n1\",\"--pretty=%H\"]
[2014-07-15 13:34:32 Pacific Daylight Time] call: git [\"--git-dir=c:\\data\\annex\\.git\",\"--work-tree=c:\\data\\annex\",\"-c\",\"core.bare=false\",\"show-ref\",\"--verify\",\"-q\",\"refs/remotes/server/synced/master\"]
[2014-07-15 13:34:32 Pacific Daylight Time] read: git [\"--git-dir=c:\\data\\annex\\.git\",\"--work-tree=c:\\data\\annex\",\"-c\",\"core.bare=false\",\"log\",\"refs/heads/synced/master..refs/remotes/server/synced/master\",\"-n1\",\"--pretty=%H\"]
[2014-07-15 13:34:32 Pacific Daylight Time] Pusher: pushing to [Remote { name =\"server\" }]
[2014-07-15 13:34:32 Pacific Daylight Time] call: git [\"--git-dir=c:\\data\\annex\\.git\",\"--work-tree=c:\\data\\annex\",\"-c\",\"core.bare=false\",\"push\",\"server\",\"+git-annex:synced/git-annex\",\"annex/direct/master:synced/master\"]
ssh: Could not resolve hostname annex: hostname nor servname provided, or not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
[2014-07-15 13:34:34 Pacific Daylight Time] read: git [\"--git-dir=c:\\data\\annex\\.git\",\"--work-tree=c:\\data\\annex\",\"-c\",\"core.bare=false\",\"push\",\"server\",\"master\"]
ssh: Could not resolve hostname annex: hostname nor servname provided, or not known

[2014-07-15 13:34:35 Pacific Daylight Time] RemoteControl: DISCONNECTED ssh://test@annex/srv/data/Shared
[2014-07-15 13:34:35 Pacific Daylight Time] RemoteControl: fromList []
[2014-07-15 13:34:36 Pacific Daylight Time] Pusher: fallback pushing to [Remote { name =\"server\" }]
[2014-07-15 13:34:36 Pacific Daylight Time] call: git [\"--git-dir=c:\\data\\annex\\.git\",\"--work-tree=c:\\data\\annex\",\"-c\",\"core.bare=false\",\"push\",\"server\",\"+git-annex:refs/synced/91347741-60cd-4bca-a64a-90adee8d1910/git-annex\",\"refs/heads/annex/direct/master:refs/synced/91347741-60cd-4bca-a64a-90adee8d1910/annex/direct/master\"]
ssh: Could not resolve hostname annex: hostname nor servname provided, or not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
[2014-07-15 13:34:51 Pacific Daylight Time] chat: ssh [\"-T\",\"test@annex\",\"git-annex-shell 'notifychanges' '/srv/data/Shared' --uuid df1c0b77-f9b6-4ed7-96f3-3e7f010c4fb6\"]
ssh: Could not resolve hostname annex: hostname nor servname provided, or not known

[2014-07-15 13:35:03 Pacific Daylight Time] RemoteControl: DISCONNECTED ssh://test@annex/srv/data/Shared
[2014-07-15 13:35:03 Pacific Daylight Time] RemoteControl: fromList []
[2014-07-15 13:35:35 Pacific Daylight Time] chat: ssh [\"-T\",\"test@annex\",\"git-annex-shell 'notifychanges' '/srv/data/Shared' --uuid df1c0b77-f9b6-4ed7-96f3-3e7f010c4fb6\"]
ss[2014-07-15 13:35:47 Pacific Daylight Time] RemoteControl: DISCONNECTED ssh://test@annex/srv/data/Shared
[2014-07-15 13:35:47 Pacific Daylight Time] RemoteControl: fromList []
[2014-07-15 13:36:51 Pacific Daylight Time] chat: ssh [\"-T\",\"test@annex\",\"git-annex-shell 'notifychanges' '/srv/data/Shared' --uuid df1c0b77-f9b6-4ed7-96f3-3e7f010c4fb6\"]
ssh: Could not resolve hostname annex: hostname nor servname provided, or not known

[2014-07-15 13:37:02 Pacific Daylight Time] RemoteControl: DISCONNECTED ssh://test@annex/srv/data/Shared
[2014-07-15 13:37:02 Pacific Daylight Time] RemoteControl: fromList []
[2014-07-15 13:39:10 Pacific Daylight Time] chat: ssh [\"-T\",\"test@annex\",\"git-annex-shell 'notifychanges' '/srv/data/Shared' --uuid df1c0b77-f9b6-4ed7-96f3-3e7f010c4fb6\"]
ssh: Co[2014-07-15 13:39:21 Pacific Daylight Time] RemoteControl: DISCONNECTED ssh://test@annex/srv/data/Shared
[2014-07-15 13:39:21 Pacific Daylight Time] RemoteControl: fromList []
[2014-07-15 13:43:37 Pacific Daylight Time] chat: ssh [\"-T\",\"test@annex\",\"git-annex-shell 'notifychanges' '/srv/data/Shared' --uuid df1c0b77-f9b6-4ed7-96f3-3e7f010c4fb6\"]
ssh: Could not resolve hostname an[2014-07-15 13:43:49 Pacific Daylight Time] RemoteControl: DISCONNECTED ssh://test@annex/srv/data/Shared
[2014-07-15 13:43:49 Pacific Daylight Time] RemoteControl: fromList []

(Recording state in git...)

git-annex: <stdin>: hGetLine: invalid argument (Bad file descriptor)
failed
git-annex: assistant: 1 failed
\"\"\"]]

Maybe it's something different but what could it be?

"""]]


code:
[[!comment format=mdwn
 username="divB"
 ip="128.12.90.218"
 subject="comment 3"
 date="2014-07-15T22:02:41Z"
 content="""
Update: You are right. It has nothing to do with the connections. I created a new repository without any sync. Same here:

    [2014-07-15 13:53:33 Pacific Daylight Time] main: starting assistant version 5.20140715-g622a376
    [2014-07-15 13:53:33 Pacific Daylight Time] Cronner: You should enable consistency checking to protect your data.
    (scanning...) [2014-07-15 13:53:38 Pacific Daylight Time] Watcher: Performing startup scan
    (started...)

    git-annex: <stdin>: hGetLine: invalid argument (Bad file descriptor)
    failed
    git-annex: assistant: 1 failed

It always takes exactly 10 minutes until it dies (you see is started 13:53:33 and it died at 14:03:33 (=modification of daemon.log).
So this is reproduceable ...

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 4"
 date="2014-07-16T18:18:11Z"
 content="""
Hmm, there are two places where the assistant happens to hard code a 10 minute time period. But one is 10 minutes from starting to configure a ssh remote in the webapp, not from start. And the other is the watcher's afterLastDaemonRun check, which only happens when a symlink appears in the repository (which shouldn't happen on windows and wouldn't necessarily happen 10 minutes after start).

Fabulous .. I just reproduced the crash!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 5"
 date="2014-07-16T18:21:24Z"
 content="""
A full log in an empty repo of the crash:

<pre>
[2014-07-16 14:00:53 Atlantic Standard Time] read: git [\"--git-dir=C:\\WINDOWS\\TT\\.git\",\"--work-tree=C:\\WINDOWS\\TT\",\"-c\",\"core.bare=false\",\"show-ref\",\"git-annex\"]
[2014-07-16 14:00:54 Atlantic Standard Time] read: git [\"--git-dir=C:\\WINDOWS\\TT\\.git\",\"--work-tree=C:\\WINDOWS\\TT\",\"-c\",\"core.bare=false\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
[2014-07-16 14:00:54 Atlantic Standard Time] read: git [\"--git-dir=C:\\WINDOWS\\TT\\.git\",\"--work-tree=C:\\WINDOWS\\TT\",\"-c\",\"core.bare=false\",\"log\",\"refs/heads/git-annex..9c07d1385e80dbc6044814357e9097fd231b8c99\",\"-n1\",\"--pretty=%H\"]
[2014-07-16 14:00:54 Atlantic Standard Time] chat: git [\"--git-dir=C:\\WINDOWS\\TT\\.git\",\"--work-tree=C:\\WINDOWS\\TT\",\"-c\",\"core.bare=false\",\"cat-file\",\"--batch\"]
[2014-07-16 14:00:54 Atlantic Standard Time] logging to C:\WINDOWS\TT\.git\annex\daemon.log
[2014-07-16 14:00:54 Atlantic Standard Time] main: starting assistant version 5.20140714-gf892329
[2014-07-16 14:00:54 Atlantic Standard Time] read: git [\"--git-dir=C:\\WINDOWS\\TT\\.git\",\"--work-tree=C:\\WINDOWS\\TT\",\"-c\",\"core.bare=false\",\"show-ref\",\"git-annex\"]
[2014-07-16 14:00:54 Atlantic Standard Time] read: git [\"--git-dir=C:\\WINDOWS\\TT\\.git\",\"--work-tree=C:\\WINDOWS\\TT\",\"-c\",\"core.bare=false\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
[2014-07-16 14:00:54 Atlantic Standard Time] read: git [\"--git-dir=C:\\WINDOWS\\TT\\.git\",\"--work-tree=C:\\WINDOWS\\TT\",\"-c\",\"core.bare=false\",\"log\",\"refs/heads/git-annex..9c07d1385e80dbc6044814357e9097fd231b8c99\",\"-n1\",\"--pretty=%H\"]
[2014-07-16 14:00:54 Atlantic Standard Time] chat: git [\"--git-dir=C:\\WINDOWS\\TT\\.git\",\"--work-tree=C:\\WINDOWS\\TT\",\"-c\",\"core.bare=false\",\"cat-file\",\"--batch\"]
[2014-07-16 14:00:54 Atlantic Standard Time] read: git [\"--git-dir=C:\\WINDOWS\\TT\\.git\",\"--work-tree=C:\\WINDOWS\\TT\",\"-c\",\"core.bare=false\",\"ls-files\",\"--stage\",\"-z\",\"--\",\"C:\\WINDOWS\\TT\"]
[2014-07-16 14:00:54 Atlantic Standard Time] Cronner: You should enable consistency checking to protect your data.
[2014-07-16 14:00:54 Atlantic Standard Time] SanityCheckerStartup: no index file; restaging
[2014-07-16 14:00:54 Atlantic Standard Time] read: git [\"--git-dir=C:\\WINDOWS\\TT\\.git\",\"--work-tree=C:\\WINDOWS\\TT\",\"-c\",\"core.bare=false\",\"ls-files\",\"--stage\",\"-z\",\"--\",\"C:\\WINDOWS\\TT\"]
[2014-07-16 14:00:54 Atlantic Standard Time] chat: git-annex [\"remotedaemon\"]
[2014-07-16 14:00:54 Atlantic Standard Time] read: git [\"--git-dir=C:\\WINDOWS\\TT\\.git\",\"--work-tree=C:\\WINDOWS\\TT\",\"-c\",\"core.bare=false\",\"ls-tree\",\"--full-tree\",\"-z\",\"--\",\"refs/heads/git-annex\",\"uuid.log\",\"remote.log\",\"trust.log\",\"group.log\",\"numcopies.log\",\"schedule.log\",\"preferred-content.log\",\"required-content.log\",\"group-preferred-content.log\"]
[2014-07-16 14:00:54 Atlantic Standard Time] Merger: watching C:\WINDOWS\TT\.git\refs
[2014-07-16 14:00:54 Atlantic Standard Time] TransferWatcher: watching for transfers
(scanning...) [2014-07-16 14:00:54 Atlantic Standard Time] Watcher: Performing startup scan
[2014-07-16 14:00:54 Atlantic Standard Time] read: git [\"--git-dir=C:\\WINDOWS\\TT\\.git\",\"--work-tree=C:\\WINDOWS\\TT\",\"-c\",\"core.bare=false\",\"ls-files\",\"--deleted\",\"-z\",\"--\",\"C:\\WINDOWS\\TT\"]
(started...) [2014-07-16 14:00:54 Atlantic Standard Time] Watcher: watching .

git-annex: <stdin>: hGetLine: invalid argument (Bad file descriptor)
failed
git-annex: assistant: 1 failed
</pre>

The two blank lines just before the crash seem to be significant somehow; they are not output at the beginning, so must occur just before it crashes.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 6"
 date="2014-07-16T18:38:51Z"
 content="""
It appears to be the remoteResponderThread crashing (based on adding debugging around all hGetLine call sites).
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 7"
 date="2014-07-16T20:18:50Z"
 content="""
Well, it seems not to be the RemoteControl thread after all. I made that thread start the remotedaemon and then not read from it at all, and the crash still happens. Also, the remotedaemon itself seems to run forever ok.

(This is sort of a relief, because it should not be possible for a crash of a named thread like RemoteControl to take down the whole assistant!)

The hGetLine must be being called by some other library function.

Hmm... Could it perhaps be waitForTermination? On windows that ends up calling getLine in a loop. A 600 second loop!

Update: Seems so. With a 60 second loop, it crashes after 60 seconds. With the 2 newlines happening just before.

Also, it makes sense now why this is a new crash. Before the assistant ran with stdio connected to the terminal, but recently on windows it got hooked up to the debug log, and stdin is closed now.

I still don't understand where the 2 newlines come from (possibly output by something while git-annex is crashing?).. But have confirmed I've fixed the problem now.
"""]]


code:
[[!comment format=mdwn
 username="divB"
 ip="204.17.143.10"
 subject="comment 8"
 date="2014-07-16T23:53:39Z"
 content="""
Really great, seems to be fixed in the current version (at least it's running for 1h now). Thank you!!
"""]]


code:
Windows has an unfortunate handling of time zones, which means that when
the time zone is changed (or DST changes), the timestamps of files also
appear to change.

This means that after such a change, git-annex will see new mtimes, and
want to re-checksum every file in the repo.

[[!tag confirmed]]

> Update: Actually, I seem to have been getting confused by behavior of
> cygwin terminal setting TZ. That indeed led to timestamp changes when the
> time zone changed. I have made git-annex unset TZ to avoid this.
>
> Without TZ set, time stamps are actually stable across time zone changes.
> Ie, a simple program to read the time stamp of a file and print it
> always shows the same thing, before and after a timezone change.
>
> However, and here's where it gets truely ghastly: A program that stats a
> file in a loop will see its timestamp change when the timezone changes.
> I suspect this might be a bug in the Haskell RTS caching something it
> should not. Stopping and re-running the program gets back to the
> original timestamp.
>
> I have not tested DST changes, but it's hard to imagine it being any
> worse than the above behavior.
>
> So, that's insane then. We can't trust timestamps to be stable on windows
> when git-annex is running for a long period of time. --[[Joey]]
>
> > [[fixed|done]], using the inode sentinal file to detect when windows
> > has lost its mind, and calculating its delta from insanity. --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 1"
 date="2014-06-11T19:09:17Z"
 content="""
Rather than getting the timezone, another approach might be to look at the inode sentinal file. Its timestamp will also appear to have changed. If the delta is exactly some number of hours, and the inode sential's other data is unchanged, a Windows-specific hack could apply that same delta to all inode cache timestamps.

Except, time zones are not all actually on hour boundaries. Some are half hours, some may be 15 minutes, and next week some crazy country might legislate a 3 minute delta for all I know.

Well, could just say if the inode sentinal's mtime has changed at all (delta < 3600 seconds), and it's otherwise unchanged, and we're on windows, assume this is a time zone change. When would that fail? Only if the repository is copied to someplace, and the mtime is not preserved.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 2"
 date="2014-06-11T19:51:15Z"
 content="""
Note that multiple time zone changes complicate this. I think that means that the delta can't be simply applied when comparing inode caches. Instead, probably it needs to be applied when generating inode caches.

A scenario:

1. Time zone is at +1h when the inode sential is written.
2. Time zone changes to +2h
3. File F is added (with a current timestamp of T)
4. Time zone changes to +5h

I am a little confused by which way windows moves the timestamps when the time zone changes. Let's assume I might get the sign wrong.

Let F's timestamp after step 4, F4 = T+-3h.

Let the delta after step 4, D4 = +-4h
And, let the delta after step 2, D2 = +-1h

If step 3 writes the current timestamp to the inode cache, then the cache still has T in it after step 4. F4+D4 /= T (T +-3h +-4h /= T). So comparison doesn't work.

If instead the current delta is applied when generating inode caches (both for storing on disk, and for immediate comparison), then the inode cache will have T+D2 in it. Then after step 4, generating a new inode cache for F will yield F4+D4. So, does F4+D4 == T+D2? T +-3h +-4h == T +-1h YES!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 3"
 date="2014-06-11T22:11:38Z"
 content="""
I have implemented this on the loststamp git branch. It seems to work!

However, it has a big problem:

    If the timezone changes while the assistant (or a long-running command)
    runs, it won't notice, since it only checks the inode cache once, and
    so will use the old delta for all new inode caches it generates for new
    files it's added. Which will result in them seeming changed the next time
    it runs.

So, it would be really nice to be able to check the actual timezone instead.
But I suppose I can make the assistant poll the inode cache file, or check
it when adding a new file, or something like that. Bleagh.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 4"
 date="2014-06-11T22:54:48Z"
 content="""
Getting the actual time zone on windows works better if you unset TZ first.

But, a haskell program that polls the time zone fails to notice when it's changed. It only notices after being restarted. I have contacted the time library maintainer about this.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 5"
 date="2014-06-11T23:02:28Z"
 content="""
I've developed a fix for the time library. This patch has been sent to the author, hopefully it will get applied and then I can use getCurrentTImeZone. Note that git-annex would need to unset TZ first, which might be hard on windows.

<pre>
diff --git a/cbits/HsTime.c b/cbits/HsTime.c
index cfafb27..86ca92a 100644
--- a/cbits/HsTime.c
+++ b/cbits/HsTime.c
@@ -8,6 +8,7 @@ long int get_current_timezone_seconds (time_t t,int* pdst,char const* * pname)
     tzset();
     struct tm* ptm = localtime_r(&t,&tmd);
 #else
+    tzset();
     struct tm* ptm = localtime(&t);
 #endif
     if (ptm)
</pre>
"""]]


code:
### Please describe the problem.
On a default, clean install on Windows, annex get over HTTP fails, as neither wget nor curl have all necessary DLLs.

### What steps will reproduce the problem?
1. Install git from git-scm.org, and git-annex from git-annex.branchable.com.
2. Don't install, or remove if installed, Cygwin.
3. clone an annex over HTTP
4. attempt to 'annex get' a file
5. get dialog box, explaining missing DLLs.

### What version of git-annex are you using? On what operating system?
Windows 7.

### Please provide any additional information below.
Installing Cygwin and adding to path seems to fix issue.

[[!tag moreinfo]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.254.222"
 subject="comment 1"
 date="2013-07-07T17:47:54Z"
 content="""
The Windows installer includes numerous DLLs from Cygwin. I have tested it in a non-cygwin environment and it seems to work for me, and for other users. At least generally.. perhaps it *is* missing something used by wget or curl?

Can you please provide the names of the DLLs that seem to be missing?
"""]]


code:
[[!tag moreinfo]]

### Please describe the problem.
git annex get on a clone of a repository created on Windows fails.

### What steps will reproduce the problem?
On Windows:
1. Create an annex using git annex init

2. git add files

3. git annex sync (because I forgot about commit)

On Linux:

1. Mount the Windows annex via SMB (mine was read only, but probably won't matter)

2. git clone the folder (works fine)

3. git annex get (fails, due to Windows drive letter)

### What version of git-annex are you using? On what operating system?

*Windows:*

git-annex version: 4.20130802-g0a52f02

build flags: Pairing Testsuite S3 WebDAV DNS

*Linux: (Debian wheezy on ARMel)*

git-annex version: 3.20120629

local repository version: 3

default repository version: 3

supported repository versions: 3

upgrade supported from repository versions: 0 1 2

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

#output of git annex get on the Linux side.

get Purchased/Caravan Palace/Panic/01 Queens.mp3 (not available)
  Try making some of these repositories available:
  	d6c78593-79d2-455a-8202-7be2133e2d48 -- ICARUS:E:\music
failed
get Purchased/Caravan Palace/Panic/02 Maniac.mp3 (not available)
  Try making some of these repositories available:
  	d6c78593-79d2-455a-8202-7be2133e2d48 -- ICARUS:E:\music
failed
get Purchased/Caravan Palace/Panic/03 The Dirty Side of the Street.mp3 (not available)
  Try making some of these repositories available:
  	d6c78593-79d2-455a-8202-7be2133e2d48 -- ICARUS:E:\music
failed
get Purchased/Caravan Palace/Panic/04 12 juin 3049.mp3 (not available)
  Try making some of these repositories available:
  	d6c78593-79d2-455a-8202-7be2133e2d48 -- ICARUS:E:\music
failed
get Purchased/Caravan Palace/Panic/05 Rock It for Me.mp3 (not available)
  Try making some of these repositories available:
  	d6c78593-79d2-455a-8202-7be2133e2d48 -- ICARUS:E:\music
failed
get Purchased/Caravan Palace/Panic/06 Clash.mp3 (not available)
  Try making some of these repositories available:
  	d6c78593-79d2-455a-8202-7be2133e2d48 -- ICARUS:E:\music
failed
get Purchased/Caravan Palace/Panic/07 Newbop.mp3 (not available)
  Try making some of these repositories available:
  	d6c78593-79d2-455a-8202-7be2133e2d48 -- ICARUS:E:\music
failed
get Purchased/Caravan Palace/Panic/08 Glory of Nelly.mp3 (not available)
  Try making some of these repositories available:
  	d6c78593-79d2-455a-8202-7be2133e2d48 -- ICARUS:E:\music
failed
get Purchased/Caravan Palace/Panic/09 Dramophone.mp3 (not available)
  Try making some of these repositories available:
  	d6c78593-79d2-455a-8202-7be2133e2d48 -- ICARUS:E:\music
failed
get Purchased/Caravan Palace/Panic/10 Cotton Heads.mp3 (not available)
  Try making some of these repositories available:
  	d6c78593-79d2-455a-8202-7be2133e2d48 -- ICARUS:E:\music
failed
get Purchased/Caravan Palace/Panic/11 Panic.mp3 (not available)
  Try making some of these repositories available:
  	d6c78593-79d2-455a-8202-7be2133e2d48 -- ICARUS:E:\music
failed
get Purchased/Caravan Palace/Panic/12 Pirates.mp3 (not available)
  Try making some of these repositories available:
  	d6c78593-79d2-455a-8202-7be2133e2d48 -- ICARUS:E:\music
failed
get Purchased/Caravan Palace/Panic/13 Beatophone.mp3 (not available)
  Try making some of these repositories available:
  	d6c78593-79d2-455a-8202-7be2133e2d48 -- ICARUS:E:\music
failed
get Purchased/Caravan Palace/Panic/14 Sydney.mp3 (not available)
  Try making some of these repositories available:
  	d6c78593-79d2-455a-8202-7be2133e2d48 -- ICARUS:E:\music
failed
get Purchased/Caravan Palace/Panic/Thumbs.db (not available)
  Try making some of these repositories available:
  	d6c78593-79d2-455a-8202-7be2133e2d48 -- ICARUS:E:\music

And the (broken) symlinks:

lrwxrwxrwx 1 voltagex voltagex 198 Aug 11 23:05 01 Queens.mp3 -> .git/annex/objects/5P/vq/SHA256E-s9886015--dfb8f452d47f997a9141de3f152de375fad1d5ccb4a20d5c022064a630eaba88.mp3/SHA256E-s9886015--dfb8f452d47f997a9141de3f152de375fad1d5ccb4a20d5c022064a630eaba88.mp3
lrwxrwxrwx 1 voltagex voltagex 200 Aug 11 23:05 02 Maniac.mp3 -> .git/annex/objects/J0/WV/SHA256E-s10116938--7ce35f4a54c4f74bd2e3813445cb3a07fae76b2854df06ab00876cd0067f34a5.mp3/SHA256E-s10116938--7ce35f4a54c4f74bd2e3813445cb3a07fae76b2854df06ab00876cd0067f34a5.mp3
lrwxrwxrwx 1 voltagex voltagex 198 Aug 11 23:05 03 The Dirty Side of the Street.mp3 -> .git/annex/objects/xz/4F/SHA256E-s8812949--9e15761ae8c6231d11d1f5a67de8a89142e2e4017b43b8ffac1088fd71842ca7.mp3/SHA256E-s8812949--9e15761ae8c6231d11d1f5a67de8a89142e2e4017b43b8ffac1088fd71842ca7.mp3
lrwxrwxrwx 1 voltagex voltagex 198 Aug 11 23:05 04 12 juin 3049.mp3 -> .git/annex/objects/Kp/k6/SHA256E-s8236133--3a6465bea3272ae7a22b77b67e96a91ae389390cc95642fb1456ce80e3313033.mp3/SHA256E-s8236133--3a6465bea3272ae7a22b77b67e96a91ae389390cc95642fb1456ce80e3313033.mp3
lrwxrwxrwx 1 voltagex voltagex 198 Aug 11 23:05 05 Rock It for Me.mp3 -> .git/annex/objects/Wk/Qv/SHA256E-s7701150--8197e499408e64601157df95927ac6b09bd37834a41eceecc5d12667bd7c66a4.mp3/SHA256E-s7701150--8197e499408e64601157df95927ac6b09bd37834a41eceecc5d12667bd7c66a4.mp3
lrwxrwxrwx 1 voltagex voltagex 200 Aug 11 23:05 06 Clash.mp3 -> .git/annex/objects/ZQ/P4/SHA256E-s10201572--c2a1a8892fcaaab1ed59ca7f5ab9d45f0c3bb3c6d6450b95228039b9e8d7a0b4.mp3/SHA256E-s10201572--c2a1a8892fcaaab1ed59ca7f5ab9d45f0c3bb3c6d6450b95228039b9e8d7a0b4.mp3
lrwxrwxrwx 1 voltagex voltagex 198 Aug 11 23:05 07 Newbop.mp3 -> .git/annex/objects/Fk/wV/SHA256E-s6850587--b5ba0347f6a09a7ff8bec2dd1fbe8076fdff645d4bc1909ddeadfc035bf19fda.mp3/SHA256E-s6850587--b5ba0347f6a09a7ff8bec2dd1fbe8076fdff645d4bc1909ddeadfc035bf19fda.mp3
lrwxrwxrwx 1 voltagex voltagex 198 Aug 11 23:05 08 Glory of Nelly.mp3 -> .git/annex/objects/Pm/K3/SHA256E-s9002048--18149581cbfc3b4e3e72b784777fe34c53f1580e78e97f66058be0b1eb40e809.mp3/SHA256E-s9002048--18149581cbfc3b4e3e72b784777fe34c53f1580e78e97f66058be0b1eb40e809.mp3
lrwxrwxrwx 1 voltagex voltagex 198 Aug 11 23:05 09 Dramophone.mp3 -> .git/annex/objects/k3/Jf/SHA256E-s8199558--925117d9fc47a65e8e5324f3d0638a3c24bf51fd6c0b5d8ac2f63951c893cc48.mp3/SHA256E-s8199558--925117d9fc47a65e8e5324f3d0638a3c24bf51fd6c0b5d8ac2f63951c893cc48.mp3
lrwxrwxrwx 1 voltagex voltagex 198 Aug 11 23:05 10 Cotton Heads.mp3 -> .git/annex/objects/9f/32/SHA256E-s8801425--c6926238c0b1a3bbea1a5d17841ceac591e53e223e4c4c45a2077cabffc85d81.mp3/SHA256E-s8801425--c6926238c0b1a3bbea1a5d17841ceac591e53e223e4c4c45a2077cabffc85d81.mp3
lrwxrwxrwx 1 voltagex voltagex 198 Aug 11 23:05 11 Panic.mp3 -> .git/annex/objects/XF/WF/SHA256E-s9833770--5e837c7fa3ec096f7c0507efbcf398067029749f8a0fc77a2badf864b9ffbb7c.mp3/SHA256E-s9833770--5e837c7fa3ec096f7c0507efbcf398067029749f8a0fc77a2badf864b9ffbb7c.mp3
lrwxrwxrwx 1 voltagex voltagex 198 Aug 11 23:05 12 Pirates.mp3 -> .git/annex/objects/7Z/jz/SHA256E-s8017742--5fd49b2d89577266d2fb740e7e7def9338475af90c2ca99f9d6d513465b2bfac.mp3/SHA256E-s8017742--5fd49b2d89577266d2fb740e7e7def9338475af90c2ca99f9d6d513465b2bfac.mp3
lrwxrwxrwx 1 voltagex voltagex 198 Aug 11 23:05 13 Beatophone.mp3 -> .git/annex/objects/81/xv/SHA256E-s9339544--a1eb8404ecf0503b9f635378fec4e2c95ec08bb1428c2cd4c0cedf492811577d.mp3/SHA256E-s9339544--a1eb8404ecf0503b9f635378fec4e2c95ec08bb1428c2cd4c0cedf492811577d.mp3
lrwxrwxrwx 1 voltagex voltagex 198 Aug 11 23:05 14 Sydney.mp3 -> .git/annex/objects/3K/99/SHA256E-s8459731--4ff44b25c912e914c79124ff9074c576c0024152442fc96c9bad65f5a50a40d9.mp3/SHA256E-s8459731--4ff44b25c912e914c79124ff9074c576c0024152442fc96c9bad65f5a50a40d9.mp3
lrwxrwxrwx 1 voltagex voltagex 192 Aug 11 23:05 Thumbs.db -> .git/annex/objects/28/Zv/SHA256E-s85504--3604669cd3a55234516191eb4f19434829c1634d6dd69a9981185f095d2bbaba.db/SHA256E-s85504--3604669cd3a55234516191eb4f19434829c1634d6dd69a9981185f095d2bbaba.db

# End of transcript or log.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlzWwnBfgJrkhPQakBo6DbPXutJIVDHkj0"
 nickname="Adam"
 subject="comment 1"
 date="2013-08-11T14:15:40Z"
 content="""
Confirming the same errors happen with git-annex version: 4.20130802 from sid.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.63"
 subject="bad bug report title 101"
 date="2013-08-24T19:17:29Z"
 content="""
I don't understand why you think the problem has something to do with Windows drive letters. There are no Windows drive letters in the symlinks you show. The only place I see any Windows drive letter is in the descripton of the remote that `git annex get` displays when it fails to get the file. That description is purely informative, it's not a path that git-annex is trying to use.

I'd suggest that you run `git annex get --debug` to see if it is doing anything obviously wrong. The mostly likely culprit is your SMB setup, which I am not going to be able to replicate.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlzWwnBfgJrkhPQakBo6DbPXutJIVDHkj0"
 nickname="Adam"
 subject="comment 3"
 date="2013-08-26T06:56:33Z"
 content="""
You're correct. I can see in .git/config that the remote references z:\ which of course will break on the Linux side. Maybe this is a case of the error messages not quite telling me the right thing?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.2.134"
 subject="comment 4"
 date="2013-09-13T19:29:25Z"
 content="""
I'd suggest that you run git annex get --debug to see if it is doing anything obviously wrong. The mostly likely culprit is your SMB setup, which I am not going to be able to replicate.

"""]]


code:
### Please describe the problem.
On a Windows machine, I try to drop some files from an annex. These files exist in the origin annex. However, the drop fails, claiming it can't verify the existence of the remote copies.

### What steps will reproduce the problem?
1. Create annex, serve git repo over HTTP.
2. clone repo. "git annex get ."
3. "git annex drop <file>". This will fail.

### What version of git-annex are you using? On what operating system?
Windows:  4.20130709-gea6fdc7
Debian: 3.20120629

### Please provide any additional information below.

[[!format sh """
git annex -vd drop file2.txt
[<datetime>] read: git ["--git-dir=C:\\Users\\<user>\\test_annex_2\\.git","--work-tree=C:\\Users\\<user>\\test_annex_2","ls-files","--cached","-z","--","file2.txt"]
[<datetime>] chat: git ["--git-dir=C:\\Users\\<user>\\test_annex_2\\.git","--work-tree=C:\\Users\\<user>\\test_annex_2","check-attr","-z","--stdin","annex.backend","annex.numcopies","--"]
drop file2.txt [<datetime>] read: git ["--git-dir=C:\\Users\\<user>\\test_annex_2\\.git","--work-tree=C:\\Users\\<user>\\test_annex_2","show-ref","git-annex"]
[<datetime>] read: git ["--git-dir=C:\\Users\\<user>\\test_annex_2\\.git","--work-tree=C:\\Users\\<user>\\test_annex_2","show-ref","--hash","refs/heads/git-annex"]
[<datetime>] read: git ["--git-dir=C:\\Users\\<user>\\test_annex_2\\.git","--work-tree=C:\\Users\\<user>\\test_annex_2","log","refs/heads/git-annex..37bdd895d74131fddbdb8d2e6ae707ee2097598e","--oneline","-n1"]
[<datetime>] read: git ["--git-dir=C:\\Users\\<<user>>\\test_annex_2\\.git","--work-tree=C:\\Users\\<user>\\test_annex_2","log","refs/heads/git-annex..f0ba6f30c63a9d586642d828c37613894191931e","--oneline","-n1"]
[<datetime>] chat: git ["--git-dir=C:\\Users\\<user>\\test_annex_2\\.git","--work-tree=C:\\Users\\<user>\\test_annex_2","cat-file","--batch"]
[<datetime>] read: curl ["-s","--head","-L","http://192.168.0.8:8000/test_annex_2/.git/annex/objects/738/e2a/SHA256-s27--4fc887746d7ea6c9574f6735bd3dfd9b0485b30acaaf0727658cd1796991dd2d/SHA256-s27--4fc887746d7ea6c9574f6735bd3dfd9b0485b30acaaf0727658cd1796991dd2d","-w","%{http_code}"]
(unsafe)
  Could only verify the existence of 0 out of 1 necessary copies

  Rather than dropping this file, try using: git annex move

  (Use --force to override this check, or adjust annex.numcopies.)
failed
git-annex: drop: 1 failed
"""]]

the given curl command fails with a 404, because the remote annex isn't using the 3-character hash directory scheme, but the 2-character one, and it seems that 'annex drop' doesn't check both schemes (unlike annex get).

> Ah, the http remote is a non-bare repository, that's why.
> Ok, [[fixed|done]]! --[[Joey]]


code:
### Please describe the problem.

When the assistant connects the second time during adding a ssh remote, it assumes port 22 when different port is given.

### What steps will reproduce the problem?

Adding a ssh git remote with non-standard port.

### What version of git-annex are you using? On what operating system?

GNU/Linux, Ubuntu 12.10, amd64, git-annex 4.20130627-g728b27a

Happens on android as well.

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

Error message (nothing related in daemon.log):

Failed to make repository

Something went wrong setting up the repository on the remote server.

Transcript:

ssh: connect to host xxxxx port 22: Connection refused

# End of transcript or log.
"""]]

> [[Fixed|done]]; this bug was introcuded in version
> 4.20130627 while making some other ssh fixes. --[[Joey]]


code:
### Please describe the problem.
git-annex assistant is currently running.  Here is the output of **ps -A u | grep -i git**:

[[!format sh """
5457 pts/2    Z+     0:00 [git-annex] <defunct>
5510 ?        Sl     0:05 git-annex assistant
5522 ?        S      0:00 git --git-dir=/home/me/annex/.git --work-tree=/home/me/annex cat-file --batch
5573 ?        S      0:00 git --git-dir=/home/me/annex/.git --work-tree=/home/me/annex cat-file --batch
5679 ?        SN     0:00 git --git-dir=/home/me/annex/.git --work-tree=/home/me/annex check-attr -z --stdin annex.backend annex.numcopies --
5680 ?        Z      0:03 [git-annex] <defunct>
5710 ?        Z      0:00 [git-annex] <defunct>
5732 pts/2    Z+     0:00 [git-annex] <defunct>
5752 pts/2    Z+     0:00 [git-annex] <defunct>
5785 ?        Ss     0:00 ssh: .git/annex/ssh/example.net [mux]
5905 ?        Z      0:00 [git-annex] <defunct>
5923 ?        Z      0:00 [git-annex] <defunct>
6513 pts/2    Z+     0:00 [git-annex] <defunct>
6552 ?        Z      0:00 [git-annex] <defunct>
7797 ?        Z      0:00 [git-annex] <defunct>
7873 pts/2    Z      0:00 [git-annex] <defunct>
8708 pts/2    Z+     0:00 [git-annex] <defunct>
9821 ?        Z      0:00 [git-annex] <defunct>
9841 pts/2    Z+     0:00 [git-annex] <defunct>
10462 ?        Z      0:00 [git-annex] <defunct>
10522 pts/2    Z      0:00 [git-annex] <defunct>
12777 pts/2    Z+     0:00 [git-annex] <defunct>
13878 pts/2    Z+     0:00 [git-annex] <defunct>
14254 ?        Z      0:00 [git-annex] <defunct>
14276 pts/2    Z+     0:00 [git-annex] <defunct>
15932 ?        Sl     0:00 git-annex transferkeys --readfd 37 --writefd 20
16022 pts/2    Sl     0:00 git-annex transferkeys --readfd 28 --writefd 22
16079 pts/2    S      0:00 git --git-dir=/mnt/debian/home/me/annex-backup/.git --work-tree=/mnt/debian/home/me/annex-backup cat-file --batch
16081 ?        S      0:00 git --git-dir=/home/me/annex/.git --work-tree=/home/me/annex cat-file --batch
31565 pts/2    Sl+    0:20 git-annex webapp
31580 pts/2    S+     0:00 git --git-dir=/mnt/debian/home/me/annex-backup/.git --work-tree=/mnt/debian/home/me/annex-backup cat-file --batch
31590 pts/2    S+     0:00 git --git-dir=/mnt/debian/home/me/annex-backup/.git --work-tree=/mnt/debian/home/me/annex-backup cat-file --batch
31618 pts/2    S+     0:00 git --git-dir=/mnt/debian/home/me/annex-backup/.git --work-tree=/mnt/debian/home/me/annex-backup cat-file --batch
31635 ?        Sl     9:26 /usr/lib/firefox/firefox /mnt/debian/home/me/annex-backup/.git/annex/webapp.html
31689 pts/2    SN+    0:00 git --git-dir=/mnt/debian/home/me/annex-backup/.git --work-tree=/mnt/debian/home/me/annex-backup check-attr -z --stdin annex.backend annex.numcopies --
31812 pts/2    Z      0:00 [git-annex] <defunct>
31835 ?        Ss     0:02 ssh: .git/annex/ssh/example.net [mux]
"""]]

> [[fixed|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 1"
 date="2013-11-03T01:12:25Z"
 content="""
This is apparently the same repository where [[all_this_went_on|bugs/Unnecessary_remote_transfers]].

I don't know if this is indicative of a problem. I see more zombies than I would normally expect, but I don't know if the number is growing, or shrinking, or staying the same. You should at least look at ps -f to see which of the multiple git-annex assistant daemons you have configured to run on this machine (not a usual configuration) is the parent of the zombies.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmNu4V5fvpLlBhaCUfXXOB0MI5NXwh8SkU"
 nickname="Adam"
 subject="comment 2"
 date="2013-11-03T01:19:42Z"
 content="""
Indeed.  I've been looking forward to setting up git-annex assistant for a long time, waiting until I had more free time, and was finally pushed to do it when I had a weird problem with Dropbox.  I'm not sure what I did wrong to make such a mess, haha.

I see \"5510 git-annex assistant\" and \"31565 git-annex webapp\"--does the \"webapp\" one count as an assistant process too?
[[!format sh \"\"\"$ ps -ef | gi git
UID        PID  PPID  C STIME TTY          TIME CMD
me        3404 31565  0 17:44 pts/2    00:00:04 [git-annex] <defunct>
me        5457 31565  0 17:59 pts/2    00:00:00 [git-annex] <defunct>
me        5510     1  0 17:59 ?        00:00:06 git-annex assistant
me        5522  5510  0 17:59 ?        00:00:00 git --git-dir=/home/me/annex/.git --work-tree=/home/me/annex cat-file --batch
me        5573  5510  0 17:59 ?        00:00:00 git --git-dir=/home/me/annex/.git --work-tree=/home/me/annex cat-file --batch
me        5679  5510  0 17:59 ?        00:00:00 git --git-dir=/home/me/annex/.git --work-tree=/home/me/annex check-attr -z --stdin annex.backend annex.numcopies --
me        5680  5510  0 17:59 ?        00:00:03 [git-annex] <defunct>
me        5710  5510  0 18:00 ?        00:00:00 [git-annex] <defunct>
me        5732 31565  0 18:00 pts/2    00:00:00 [git-annex] <defunct>
me        5752 31565  0 18:00 pts/2    00:00:00 [git-annex] <defunct>
me        5785     1  0 18:01 ?        00:00:00 ssh: .git/annex/ssh/example.net [mux]
me        5905  5510  0 18:02 ?        00:00:00 [git-annex] <defunct>
me        5923  5510  0 18:02 ?        00:00:00 [git-annex] <defunct>
me        6513 31565  0 18:05 pts/2    00:00:00 [git-annex] <defunct>
me        6552  5510  0 18:06 ?        00:00:00 [git-annex] <defunct>
me        7797  5510  0 18:09 ?        00:00:00 [git-annex] <defunct>
me        7873 31565  0 18:09 pts/2    00:00:00 [git-annex] <defunct>
me        8708 31565  0 18:10 pts/2    00:00:00 [git-annex] <defunct>
me        9821  5510  0 18:12 ?        00:00:00 [git-annex] <defunct>
me        9841 31565  0 18:14 pts/2    00:00:00 [git-annex] <defunct>
me       10462  5510  0 18:23 ?        00:00:00 [git-annex] <defunct>
me       10522 31565  0 18:24 pts/2    00:00:00 [git-annex] <defunct>
me       12777 31565  0 18:34 pts/2    00:00:00 [git-annex] <defunct>
me       13878 31565  0 18:35 pts/2    00:00:00 [git-annex] <defunct>
me       14254  5510  0 18:36 ?        00:00:00 [git-annex] <defunct>
me       14276 31565  0 18:36 pts/2    00:00:00 [git-annex] <defunct>
me       15932  5510  0 18:55 ?        00:00:00 git-annex transferkeys --readfd 37 --writefd 20
me       16022 31565  0 18:55 pts/2    00:00:00 git-annex transferkeys --readfd 28 --writefd 22
me       16079 16022  0 18:55 pts/2    00:00:00 git --git-dir=/mnt/debian/home/me/annex-backup/.git --work-tree=/mnt/debian/home/me/annex-backup cat-file --batch
me       16081 15932  0 18:55 ?        00:00:00 git --git-dir=/home/me/annex/.git --work-tree=/home/me/annex cat-file --batch
me       25136 25135  0 20:15 pts/6    00:00:00 grep -i git
me       31565 31478  0 17:42 pts/2    00:00:21 git-annex webapp
me       31580 31565  0 17:42 pts/2    00:00:00 git --git-dir=/mnt/debian/home/me/annex-backup/.git --work-tree=/mnt/debian/home/me/annex-backup cat-file --batch
me       31590 31565  0 17:42 pts/2    00:00:00 git --git-dir=/mnt/debian/home/me/annex-backup/.git --work-tree=/mnt/debian/home/me/annex-backup cat-file --batch
me       31618 31565  0 17:42 pts/2    00:00:00 git --git-dir=/mnt/debian/home/me/annex-backup/.git --work-tree=/mnt/debian/home/me/annex-backup cat-file --batch
me       31635 27578  8 17:42 ?        00:13:42 /usr/lib/firefox/firefox /mnt/debian/home/me/annex-backup/.git/annex/webapp.html
me       31689 31565  0 17:42 pts/2    00:00:00 git --git-dir=/mnt/debian/home/me/annex-backup/.git --work-tree=/mnt/debian/home/me/annex-backup check-attr -z --stdin annex.backend annex.numcopies --
me       31812 31565  0 17:42 pts/2    00:00:00 [git-annex] <defunct>
me       31835     1  0 17:42 ?        00:00:02 ssh: .git/annex/ssh/example.net [mux]\"\"\"]]

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 3"
 date="2013-11-03T01:31:26Z"
 content="""
`ps fax` is much easier to read

Yes, the assistant can be started by either `git annex assistant` or, if it's not already running, `git annex webapp`
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmNu4V5fvpLlBhaCUfXXOB0MI5NXwh8SkU"
 nickname="Adam"
 subject="comment 4"
 date="2013-11-03T01:35:32Z"
 content="""
Ah, thank you.  I get lost in ps's man page.  :)

[[!format sh \"\"\"
31635 ?        Rl    16:16  \_ /usr/lib/firefox/firefox /mnt/debian/home/me/annex-backup/.git/annex/webapp.html
31565 pts/2    Sl+    0:21  |       \_ git-annex webapp
31580 pts/2    S+     0:00  |           \_ git --git-dir=/mnt/debian/home/me/annex-backup/.git --work-tree=/mnt/debian/home/me/annex-backup cat-file --batch
31590 pts/2    S+     0:00  |           \_ git --git-dir=/mnt/debian/home/me/annex-backup/.git --work-tree=/mnt/debian/home/me/annex-backup cat-file --batch
31618 pts/2    S+     0:00  |           \_ git --git-dir=/mnt/debian/home/me/annex-backup/.git --work-tree=/mnt/debian/home/me/annex-backup cat-file --batch
31689 pts/2    SN+    0:00  |           \_ git --git-dir=/mnt/debian/home/me/annex-backup/.git --work-tree=/mnt/debian/home/me/annex-backup check-attr -z --stdin annex.backend annex.numcopies --
31812 pts/2    Z      0:00  |           \_ [git-annex] <defunct>
 3404 pts/2    Z      0:04  |           \_ [git-annex] <defunct>
 5457 pts/2    Z+     0:00  |           \_ [git-annex] <defunct>
 5732 pts/2    Z+     0:00  |           \_ [git-annex] <defunct>
 5752 pts/2    Z+     0:00  |           \_ [git-annex] <defunct>
 6513 pts/2    Z+     0:00  |           \_ [git-annex] <defunct>
 7873 pts/2    Z      0:00  |           \_ [git-annex] <defunct>
 8708 pts/2    Z+     0:00  |           \_ [git-annex] <defunct>
 9841 pts/2    Z+     0:00  |           \_ [git-annex] <defunct>
10522 pts/2    Z      0:00  |           \_ [git-annex] <defunct>
12777 pts/2    Z+     0:00  |           \_ [git-annex] <defunct>
13878 pts/2    Z+     0:00  |           \_ [git-annex] <defunct>
14276 pts/2    Z+     0:00  |           \_ [git-annex] <defunct>
16022 pts/2    Sl     0:00  |           \_ git-annex transferkeys --readfd 28 --writefd 22
16079 pts/2    S      0:00  |               \_ git --git-dir=/mnt/debian/home/me/annex-backup/.git --work-tree=/mnt/debian/home/me/annex-backup cat-file --batch
26828 pts/6    S+     0:00  |       \_ grep -i git
31835 ?        Ss     0:02 ssh: .git/annex/ssh/example.net [mux]
 5510 ?        Sl     0:06 git-annex assistant
 5522 ?        S      0:00  \_ git --git-dir=/home/me/annex/.git --work-tree=/home/me/annex cat-file --batch
 5573 ?        S      0:00  \_ git --git-dir=/home/me/annex/.git --work-tree=/home/me/annex cat-file --batch
 5679 ?        SN     0:00  \_ git --git-dir=/home/me/annex/.git --work-tree=/home/me/annex check-attr -z --stdin annex.backend annex.numcopies --
 5680 ?        Z      0:03  \_ [git-annex] <defunct>
 5710 ?        Z      0:00  \_ [git-annex] <defunct>
 5905 ?        Z      0:00  \_ [git-annex] <defunct>
 5923 ?        Z      0:00  \_ [git-annex] <defunct>
 6552 ?        Z      0:00  \_ [git-annex] <defunct>
 7797 ?        Z      0:00  \_ [git-annex] <defunct>
 9821 ?        Z      0:00  \_ [git-annex] <defunct>
10462 ?        Z      0:00  \_ [git-annex] <defunct>
14254 ?        Z      0:00  \_ [git-annex] <defunct>
15932 ?        Sl     0:00  \_ git-annex transferkeys --readfd 37 --writefd 20
16081 ?        S      0:00      \_ git --git-dir=/home/me/annex/.git --work-tree=/home/me/annex cat-file --batch
 5785 ?        Ss     0:00 ssh: .git/annex/ssh/example.net [mux]    \"\"\"]]
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 5"
 date="2013-11-03T02:33:14Z"
 content="""
I believe that these zombies were all caused by switching between repository views in the webapp. I was able to reproduce 1 zombie per switch between repos.

There are a few other places where git-annex execs itself, but not many, and most of them are often used and it would be noticed if they were crypts from which pour forth the living dead. (Oddly the TV show I was thinking about watching this evening.)

<pre>
joey@darkstar:~/src/git-annex>git grep readProgramFile
Assistant/Repair.hs:		program <- readProgramFile
Assistant/Threads/Cronner.hs:	program <- liftIO $ readProgramFile
Assistant/Threads/Cronner.hs:			program <- readProgramFile
Assistant/Threads/Transferrer.hs:	program <- liftIO readProgramFile
Assistant/TransferSlots.hs:		program <- liftIO readProgramFile
Assistant/WebApp/Control.hs:		program <- readProgramFile
Assistant/WebApp/OtherRepos.hs:	program <- readProgramFile
Assistant/XMPP/Git.hs:		program <- readProgramFile
Command/Assistant.hs:	program <- readProgramFile
Config/Files.hs:readProgramFile :: IO FilePath
Config/Files.hs:readProgramFile = do
Remote/Git.hs:		program <- readProgramFile
</pre>
"""]]


code:
### Please describe the problem.

some time ago i was using the webapp bound to a dedicated port number to get around firewall reconfig. Now after some time without using the webapp i'm using it again and when i start it with

     git-annex webapp --listen=192.168.21.12:46199

it never starts but just keeps waiting forever(?)

Update:(to clarify - the following works fine but results in the "random" port "problem")

    git-annex webapp --listen=192.168.21.12

### What steps will reproduce the problem?

git-annex webapp --listen=192.168.21.12:46199

### What version of git-annex are you using? On what operating system?

version 5.20140716-g8c14ba8 on debian wheezy using your pre build static tar archive.

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

the log output is the following 3 lines

[2014-07-23 16:41:45 CEST] main: starting assistant version 5.20140716-g8c14ba8
WebApp crashed: getAddrInfo: does not exist (Name or service not known)
[2014-07-23 16:41:45 CEST] WebApp: warning WebApp crashed: getAddrInfo: does not exist (Name or service not known)

# End of transcript or log.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="24.159.78.125"
 subject="comment 1"
 date="2014-07-23T22:10:37Z"
 content="""
Support for --listen with a port was removed in version 5.20140306, since it was buggy. In particular, when the webapp creates a new repository, it needs to switch to a new port to serve that repository, so specifying a single port won't work.

Instead, when annex.listen or --listen specifies the address to listen on, `git annex webapp` will print out the url to use to open it, including the port it picked. This could be used in a script, or clicked on in the terminal to open a local browser when running the webapp on a remote host.
"""]]


code:
[[!comment format=mdwn
 username="markusk"
 ip="79.243.250.79"
 subject="comment 2"
 date="2014-07-23T23:18:37Z"
 content="""
Thank you for the info! Will add the port grep to my auth grep script as you suggested.
"""]]


code:
### Please describe the problem.

I wanted to inspect the state of an object present in a bare remote
without touching the working copy. So I issued «transferkey $KEY --from
bare-remote»; however, even for keys that are definitely present on
the remote, this would fail with «requested key is not present».

The same method, when using a non-bare repo in «--from», would succeed
under otherwise identical conditions.

### What steps will reproduce the problem?

Cf. above.

### What version of git-annex are you using? On what operating system?

git-annex 5.20140708

Linux 3.15.4

> [[done]] not a bug per comment.


code:
[[!comment format=mdwn
 username="zardoz"
 ip="134.147.14.84"
 subject="comment 1"
 date="2014-07-09T08:51:22Z"
 content="""
[[!format sh \"\"\"

# Key present on remote.
seb@vserver [ /srv/git/seb/ebooks.git ]$ ls -l annex/objects/f15/e83/WORM-s55921664-m1404679753--calibre%metadata.db/WORM-s55921664-m1404679753--calibre%metadata.db
-r--r--r-- 1 seb users 54M  6. Jul 22:49 annex/objects/f15/e83/WORM-s55921664-m1404679753--calibre%metadata.db/WORM-s55921664-m1404679753--calibre%metadata.db

# Trying to transfer key from remote.
seb [ ~/Ebooks ]$ git annex transferkey WORM-s55921664-m1404679753--calibre%metadata.db --from vserver -d
[2014-07-09 10:49:37 CEST] read: git [\"--git-dir=/home/seb/Ebooks/.git\",\"--work-tree=/home/seb/Ebooks\",\"-c\",\"core.bare=false\",\"show-ref\",\"git-annex\"]
[2014-07-09 10:49:37 CEST] read: git [\"--git-dir=/home/seb/Ebooks/.git\",\"--work-tree=/home/seb/Ebooks\",\"-c\",\"core.bare=false\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
[2014-07-09 10:49:37 CEST] read: git [\"--git-dir=/home/seb/Ebooks/.git\",\"--work-tree=/home/seb/Ebooks\",\"-c\",\"core.bare=false\",\"log\",\"refs/heads/git-annex..d2ae02b03f8b87c3465c82f96a924ac7b913fed7\",\"-n1\",\"--pretty=%H\"]
[2014-07-09 10:49:37 CEST] read: git [\"--git-dir=/home/seb/Ebooks/.git\",\"--work-tree=/home/seb/Ebooks\",\"-c\",\"core.bare=false\",\"log\",\"refs/heads/git-annex..fc2251ff897bdbc3152c98c4142dcdf77328404b\",\"-n1\",\"--pretty=%H\"]
[2014-07-09 10:49:37 CEST] read: git [\"--git-dir=/home/seb/Ebooks/.git\",\"--work-tree=/home/seb/Ebooks\",\"-c\",\"core.bare=false\",\"log\",\"refs/heads/git-annex..717b07d139b3375cab8c0d54c5dc6379a6c5cb89\",\"-n1\",\"--pretty=%H\"]
[2014-07-09 10:49:37 CEST] chat: git [\"--git-dir=/home/seb/Ebooks/.git\",\"--work-tree=/home/seb/Ebooks\",\"-c\",\"core.bare=false\",\"cat-file\",\"--batch\"]
[2014-07-09 10:49:37 CEST] read: ssh [\"-O\",\"stop\",\"-S\",\"wirrsal.net\",\"-o\",\"ControlMaster=auto\",\"-o\",\"ControlPersist=yes\",\"localhost\"]

[2014-07-09 10:49:37 CEST] read: rsync [\"--progress\",\"--inplace\",\"--perms\",\"-e\",\"'ssh' '-S' '.git/annex/ssh/wirrsal.net' '-o' 'ControlMaster=auto' '-o' 'ControlPersist=yes' '-T' 'wirrsal.net' 'git-annex-shell ''sendkey'' ''/srv/git/seb/ebooks.git'' ''WORM-s55921664-m1404679753--calibre%metadata.db'' --uuid 4316c3dc-5b6d-46eb-b780-948c717b7be5 ''--'' ''remoteuuid=a97ab0e5-320e-488b-b9e8-b9da57785b81'' ''direct=1'' ''--'''\",\"--\",\"dummy:\",\"/home/seb/Ebooks/.git/annex/tmp/WORM-s55921664-m1404679753--calibre&smetadata.db\"]
  requested key is not present
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [Receiver=3.1.1]

  rsync failed -- run git annex again to resume file transfer

\"\"\"]]
"""]]


code:
[[!comment format=mdwn
 username="zardoz"
 ip="134.147.14.84"
 subject="comment 2"
 date="2014-07-09T09:02:59Z"
 content="""
Ah, sorry. I see. So, the «%» have to be unescaped into slashes, of course. My bad.

[[!tag done]]
"""]]


code:
### Please describe the problem.

I suppose that in a direct mode repo, one might as well just «chmod -R
+w .git/annex; rm -r .git/annex», but I noticed that, when using «git
annex uninit», this will fail to remove some files in .git/annex.

### What steps will reproduce the problem?
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
$ mkdir /tmp/foo
$ cd /tmp/foo
$ git init
$ git annex init
$ echo quux > file
$ git annex add file
$ git annex sync
$ git annex uninit
unannex file ok
git-annex: /tmp/foo/.git/annex/objects/zQ/MQ/SHA256E-s3--98ea6e4f216f2fb4b69fff9b3a44842c38686ca685f3f55dc48c5d3fb1107be4/SHA256E-s3--98ea6e4f216f2fb4b69fff9b3a44842c38686ca685f3f55dc48c5d3fb1107be4.map: removeLink: permission denied (Permission denied)

# End of transcript or log.
"""]]

### What version of git-annex are you using? On what operating system?
git-annex 5.20140709-1

linux 3.15.5

### Please provide any additional information below.

> [[fixed|done]] --[[Joey]]


code:
Wow, what a great archiving system. Thank you for all your work on git annex!

### Please describe the problem.

I was using 'git annex assistant' on a brand-new annex that I created today. I had previously added about 20GB of data and a couple thousand files, mostly MP4 videos and MP3 music.

I then used regular 'mv' to add a folder containing about 20GB of music. This went well for a while—git annex assistant added two groups of files, containing roughly 700 and 1000 MP3s each. But the third group contained 4,923 files, and it's taking a really long time to import.

CPU usage is pretty consistently near 100%. According to the log, the files are being processed slowly.

### What steps will reproduce the problem?

I don't want to try to reproduce this problem until the MP3s finish being imported. I can try again later with thousands of digital photos if that would help.

### What version of git-annex are you using? On what operating system?

Version: 4.20130709.1
Build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP

Ubuntu 12.04.2 LTS

### Please provide any additional information below.

Here's the 'top' output and a snippet of the log. Let me know if you need anything else.

[[!format sh """
# CPU usage

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
  584 ...me...  20   0  776m 147m  16m S  100  0.9 181:16.87 git-annex

# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

[201ok
add music/Pop/Various/Like, Omigod! The 80s Pop Culture Box (totally)/._3-06 Words.mp3 3-0(checksum...) 7-22 14:52:14 EDT] TransferScanner: queued Upload UUID "8dbe75a4-b065-46fd-99f7-22599b2eaf06" music/Rock/Aerosmith/A Little South of Sanity/._2-08 Walk This Way.mp3 Nothing : expensive scan found missing object
[2013-07-22 14:52:34 EDT] Transferrer: Transferring: Upload UUID "8dbe75a4-b065-46fd-99f7-22599b2eaf06" music/Rock/Aerosmith/A Little South of Sanity/._2-08 Walk This Way.mp3 Nothing
[2013-07-22 14:52:34 EDT] chat: git ["--git-dir=/mnt/storage/private/annex/.git","--work-tree=/mnt/storage/private/annex","hash-object","-t","blob","-w","--stdin","--no-filters"]
[ok
add music/Pop/Various/Like, Omigod! The 80s Pop Culture Box (totally)/._4-05 Our House.mp3 2013-0(checksum...) 7-22 14:52:34 EDT] TransferWatcher: transfer starting: Upload UUID "8dbe75a4-b065-46fd-99f7-22599b2eaf06" music/Rock/Aerosmith/A Little South of Sanity/._2-08 Walk This Way.mp3 Nothing
[2013-07-22 14:52:54 EDT] TransferWatcher: transfer finishing: Transfer {transferDirection = Upload, transferUUID = UUID "8dbe75a4-b065-46fd-99f7-22599b2eaf06", transferKey = Key {keyName = "a8ddf79be61cf4a5ab3c7c8e95d8c259ceb102410dff50eb1260e7d818f8c5a8.mp3", keyBackendName = "SHA256E", keySize = Just 70, keyMtime = Nothing}}
[2013-07-22 14:52:54 EDT] chat: git ["--git-dir=/mnt/storage/private/annex/.git","--work-tree=/mnt/storage/private/annex","hash-object","-t","blob","-w","--stdin","--no-filters"]
ok
add music/Pop/Various/Like, Omigod! The 80s Pop Culture Box (totally)/7-15 Never Gonna Give You Up.mp3 (checksum...) [2013-07-22 14:52:54 EDT] read: sha256sum ["/mnt/storage/private/annex/.git/annex/tmp/7-15 Never Gonna Give You Up584.mp3"]
[2013-07-22 14:52:55 EDT] chat: git ["--git-dir=/mnt/storage/private/annex/.git","--work-tree=/mnt/storage/private/annex","hash-object","-t","blob","-w","--stdin","--no-filters"]
[ok
add music/Pop/Various/Like, Omigod! The 80s Pop Culture Box (totally)/3-09 Down Under.mp3 2013-07-(checksum...) 22 14:52:55 EDT] TransferScanner: queued Upload UUID "8dbe75a4-b065-46fd-99f7-22599b2eaf06" music/Rock/Aerosmith/A Little South of Sanity/._2-09 Dude (Looks Like A Lady).mp3 Nothing : expensive scan found missing object
[2013-07-22 14:52:55 EDT] read: sha256sum ["/mnt/storage/private/annex/.git/annex/tmp/3-09 Down Under584.mp3"]
[2013-07-22 14:52:55 EDT] Transferrer: Transferring: Upload UUID "8dbe75a4-b065-46fd-99f7-22599b2eaf06" music/Rock/Aerosmith/A Little South of Sanity/._2-09 Dude (Looks Like A Lady).mp3 Nothing
[2013-07-22 14:52:55 EDT] chat: git ["--git-dir=/mnt/storage/private/annex/.git","--work-tree=/mnt/storage/private/annex","hash-object","-t","blob","-w","--stdin","--no-filters"]
[ok
add music/Pop/Various/Like, Omigod! The 80s Pop Culture Box (totally)/7-19 Right Here Waiting.mp3 2013(checksum...) -07-22 14:52:55 EDT] TransferWatcher: transfer starting: Upload UUID "8dbe75a4-b065-46fd-99f7-22599b2eaf06" music/Rock/Aerosmith/A Little South of Sanity/._2-09 Dude (Looks Like A Lady).mp3 Nothing
[2013-07-22 14:52:55 EDT] read: sha256sum ["/mnt/storage/private/annex/.git/annex/tmp/7-19 Right Here Waiting584.mp3"]
[2013-07-22 14:52:55 EDT] TransferWatcher: transfer finishing: Transfer {transferDirection = Upload, transferUUID = UUID "8dbe75a4-b065-46fd-99f7-22599b2eaf06", transferKey = Key {keyName = "a8ddf79be61cf4a5ab3c7c8e95d8c259ceb102410dff50eb1260e7d818f8c5a8.mp3", keyBackendName = "SHA256E", keySize = Just 70, keyMtime = Nothing}}
[2013-07-22 14:52:55 EDT] chat: git ["--git-dir=/mnt/storage/private/annex/.git","--work-tree=/mnt/storage/private/annex","hash-object","-t","blob","-w","--stdin","--no-filters"]
ok
add music/Pop/Various/Like, Omigod! The 80s Pop Culture Box (totally)/._1-04 Another One Bites The Dust.mp3 (checksum...) [2013-07-22 14:53:15 EDT] chat: git ["--git-dir=/mnt/storage/private/annex/.git","--work-tree=/mnt/storage/private/annex","hash-object","-t","blob","-w","--stdin","--no-filters"]
[2013ok
add music/Pop/Various/Like, Omigod! The 80s Pop Culture Box (totally)/2-08 Hold On Loosely.mp3 -07(checksum...) -22 14:53:15 EDT] TransferScanner: queued Upload UUID "8dbe75a4-b065-46fd-99f7-22599b2eaf06" music/Rock/Aerosmith/A Little South of Sanity/._2-10 What It Takes.mp3 Nothing : expensive scan found missing object
[2013-07-22 14:53:15 EDT] read: sha256sum ["/mnt/storage/private/annex/.git/annex/tmp/2-08 Hold On Loosely584.mp3"]
[2013-07-22 14:53:15 EDT] Transferrer: Transferring: Upload UUID "8dbe75a4-b065-46fd-99f7-22599b2eaf06" music/Rock/Aerosmith/A Little South of Sanity/._2-10 What It Takes.mp3 Nothing
[2013-07-22 14:53:15 EDT] chat: git ["--git-dir=/mnt/storage/private/annex/.git","--work-tree=/mnt/storage/private/annex","hash-object","-t","blob","-w","--stdin","--no-filters"]
[ok
add music/Pop/Various/Like, Omigod! The 80s Pop Culture Box (totally)/._6-11 Kyrie.mp3 201(checksum...) 3-07-22 14:53:15 EDT] TransferWatcher: transfer starting: Upload UUID "8dbe75a4-b065-46fd-99f7-22599b2eaf06" music/Rock/Aerosmith/A Little South of Sanity/._2-10 What It Takes.mp3 Nothing
[2013-07-22 14:53:36 EDT] TransferWatcher: transfer finishing: Transfer {transferDirection = Upload, transferUUID = UUID "8dbe75a4-b065-46fd-99f7-22599b2eaf06", transferKey = Key {keyName = "a8ddf79be61cf4a5ab3c7c8e95d8c259ceb102410dff50eb1260e7d818f8c5a8.mp3", keyBackendName = "SHA256E", keySize = Just 70, keyMtime = Nothing}}
[2013-07-22 14:53:36 EDT] chat: git ["--git-dir=/mnt/storage/private/annex/.git","--work-tree=/mnt/storage/private/annex","hash-object","-t","blob","-w","--stdin","--no-filters"]
ok
add music/Pop/Various/Like, Omigod! The 80s Pop Culture Box (totally)/._5-03 I'm So Excited.mp3 (checksum...) [2013-07-22 14:53:56 EDT] chat: git ["--git-dir=/mnt/storage/private/annex/.git","--work-tree=/mnt/storage/private/annex","hash-object","-t","blob","-w","--stdin","--no-filters"]
[2013ok
add music/Pop/Various/Like, Omigod! The 80s Pop Culture Box (totally)/7-20 Roam.mp3 -07-(checksum...) 22 14:53:56 EDT] TransferScanner: queued Upload UUID "8dbe75a4-b065-46fd-99f7-22599b2eaf06" music/Rock/Aerosmith/A Little South of Sanity/._2-11 Sweet Emotion.mp3 Nothing : expensive scan found missing object
[2013-07-22 14:53:56 EDT] read: sha256sum ["/mnt/storage/private/annex/.git/annex/tmp/7-20 Roam584.mp3"]
[2013-07-22 14:53:57 EDT] Transferrer: Transferring: Upload UUID "8dbe75a4-b065-46fd-99f7-22599b2eaf06" music/Rock/Aerosmith/A Little South of Sanity/._2-11 Sweet Emotion.mp3 Nothing
[2013-07-22 14:53:57 EDT] chat: git ["--git-dir=/mnt/storage/private/annex/.git","--work-tree=/mnt/storage/private/annex","hash-object","-t","blob","-w","--stdin","--no-filters"]
[2ok
add music/Pop/Various/Like, Omigod! The 80s Pop Culture Box (totally)/._3-20 Goodbye To You.mp3 013-07(checksum...) -22 14:53:57 EDT] TransferWatcher: transfer finishing: Transfer {transferDirection = Upload, transferUUID = UUID "8dbe75a4-b065-46fd-99f7-22599b2eaf06", transferKey = Key {keyName = "a8ddf79be61cf4a5ab3c7c8e95d8c259ceb102410dff50eb1260e7d818f8c5a8.mp3", keyBackendName = "SHA256E", keySize = Just 70, keyMtime = Nothing}}
[2013-07-22 14:54:17 EDT] chat: git ["--git-dir=/mnt/storage/private/annex/.git","--work-tree=/mnt/storage/private/annex","hash-object","-t","blob","-w","--stdin","--no-filters"]
ok
add music/Pop/Various/Like, Omigod! The 80s Pop Culture Box (totally)/._7-18 Don't Worry Be Happy.mp3 (checksum...) [2013-07-22 14:54:37 EDT] chat: git ["--git-dir=/mnt/storage/private/annex/.git","--work-tree=/mnt/storage/private/annex","hash-object","-t","blob","-w","--stdin","--no-filters"]
[ok
add music/Pop/Various/Like, Omigod! The 80s Pop Culture Box (totally)/._3-04 Rock This Town.mp3 2013-(checksum...) 07-22 14:54:37 EDT] TransferScanner: queued Upload UUID "8dbe75a4-b065-46fd-99f7-22599b2eaf06" music/Rock/Aerosmith/A Little South of Sanity/1-01 Eat The Rich.mp3 Nothing : expensive scan found missing object
[2013-07-22 14:54:57 EDT] Transferrer: Transferring: Upload UUID "8dbe75a4-b065-46fd-99f7-22599b2eaf06" music/Rock/Aerosmith/A Little South of Sanity/1-01 Eat The Rich.mp3 Nothing
[2013-07-22 14:54:57 EDT] chat: git ["--git-dir=/mnt/storage/private/annex/.git","--work-tree=/mnt/storage/private/annex","hash-object","-t","blob","-w","--stdin","--no-filters"]
ok
add music/Pop/Various/Like, Omigod! The 80s Pop Culture Box (totally)/._7-13 Since You've Been Gone.mp3 [2013-(checksum...) 07-22 14:54:57 EDT] TransferWatcher: transfer finishing: Transfer {transferDirection = Upload, transferUUID = UUID "8dbe75a4-b065-46fd-99f7-22599b2eaf06", transferKey = Key {keyName = "2cb6e7b6ee77f9f98e01e942185265dfe18868503e93d78201485672e6939ab7.mp3", keyBackendName = "SHA256E", keySize = Just 6354105, keyMtime = Nothing}}
[2013-07-22 14:55:18 EDT] chat: git ["--git-dir=/mnt/storage/private/annex/.git","--work-tree=/mnt/storage/private/annex","hash-object","-t","blob","-w","--stdin","--no-filters"]
ok
add music/Pop/Various/Like, Omigod! The 80s Pop Culture Box (totally)/._2-09 Believe It Or Not (Theme From _Greatest American Hero_).mp3 (checksum...) [2013-07-22 14:55:38 EDT] chat: git ["--git-dir=/mnt/storage/private/annex/.git","--work-tree=/mnt/storage/private/annex","hash-object","-t","blob","-w","--stdin","--no-filters"]
[ok
add music/Pop/Various/Like, Omigod! The 80s Pop Culture Box (totally)/7-14 Only In My Dreams.mp3 2013(checksum...) -07-22 14:55:38 EDT] TransferScanner: queued Upload UUID "8dbe75a4-b065-46fd-99f7-22599b2eaf06" music/Rock/Aerosmith/A Little South of Sanity/1-02 Love In An Elevator.mp3 Nothing : expensive scan found missing object
[2013-07-22 14:55:38 EDT] read: sha256sum ["/mnt/storage/private/annex/.git/annex/tmp/7-14 Only In My Dreams584.mp3"]
[2013-07-22 14:55:38 EDT] Transferrer: Transferring: Upload UUID "8dbe75a4-b065-46fd-99f7-22599b2eaf06" music/Rock/Aerosmith/A Little South of Sanity/1-02 Love In An Elevator.mp3 Nothing
[2013-07-22 14:55:38 EDT] chat: git ["--git-dir=/mnt/storage/private/annex/.git","--work-tree=/mnt/storage/private/annex","hash-object","-t","blob","-w","--stdin","--no-filters"]
[2013-ok
add music/Pop/Various/Like, Omigod! The 80s Pop Culture Box (totally)/._4-08 Talking In Your Sleep.mp3 07-2(checksum...) 2 14:55:38 EDT] TransferWatcher: transfer finishing: Transfer {transferDirection = Upload, transferUUID = UUID "8dbe75a4-b065-46fd-99f7-22599b2eaf06", transferKey = Key {keyName = "7cd4b9aefb99f044c5f3b24e9890f45673a63ada3b71e7399e17eb1d710ea0f6.mp3", keyBackendName = "SHA256E", keySize = Just 7181660, keyMtime = Nothing}}

# End of transcript or log.
"""]]

[[!meta title="direct mode mappings scale badly with thousands of identical files"]]

[[!tag confirmed]]


code:
[[!comment format=mdwn
 username="http://emk.myopenid.com/"
 ip="24.2.150.84"
 subject="Interesting speedup"
 date="2013-07-22T19:45:58Z"
 content="""
I noticed that there were a lot of junk files with names of the form \"._*\" in the \"music\" directory I had added. These are created by older MacOS X systems storing Mac-related files on Unix file systems. (I think they're supposed to contain the old resource fork or something weird like that.)

So while the endless import was running, I decided to live dangerous and delete the offending files:

    find music/ -name ._\* -print0 | xargs -0 rm

A few seconds later, the 4,923 file import finished.

If I discover anything else interesting, I'll mention it. Once again, many thanks!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.1.10"
 subject="comment 2"
 date="2013-07-22T20:15:03Z"
 content="""
I doubt that removing the junk files did anything, unless perhaps it deleted all the files that it had not yet gotten around to adding, and so short-circuited the slow part of the process.

Adding a lot of files in a repository in direct mode can be slowed down some because it has to run `git hash-object` once per file to stage the symlink. It may be possible to speed this up by changing the code to write the symlink to disk in a temporary location, which would then allow it to use the single `git hash-object --batch` it keeps running, and just tell it to hash that file.

It seems likely to me though that the sha256sum it has to run on every file before adding it is responsible for more of the slowdown. After all, that has to read every file from disk (here apparently from `/mnt` which, if it's a USB device etc could be pretty slow.) You can benchmark this at home: First look at the debug log to find the start and finish times of it adding all those files. Then clear all disk caches. Then run sha256sum on every file and time that. Compare & post here. ;)

Adding to the overhead, the assistant is uploading every file it adds to the remote with uuid 8dbe75a4-b065-46fd-99f7-22599b2eaf06. Which means yet another read of the file, and more work depending on what kind of remote that is and how expensive it is to transfer to it.

So, in summary, hashing, recording in git, and backing up a lot of files is really slow. It would be good to work out which parts are the slow parts, so we can think about whether that speed is acceptable for that part.
"""]]


code:
[[!comment format=mdwn
 username="http://emk.myopenid.com/"
 ip="24.2.150.84"
 subject="That particular "Adding 4923 files" was unusually slow"
 date="2013-07-23T11:46:03Z"
 content="""
That particular add took most of an afternoon, and didn't complete until I deleted all the junk files.

I later did \"adding ~10,000 files\" and \"adding ~15,000 files\", all of which ran in an hour or so at most, even though I had added a second remote. So this isn't solely a disk I/O bottleneck.

One peculiar thing about the garbage files: There were 1,626 of them, and they were all tiny:

```
00000000  00 05 16 07 00 02 00 00  00 00 00 00 00 00 00 00  |................|
00000010  00 00 00 00 00 00 00 00  00 01 00 00 00 09 00 00  |................|
00000020  00 26 00 00 00 20 4d 50  47 33 68 6f 6f 6b 00 00  |.&... MPG3hook..|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000046
```

1,581 of these files had the SHA1 hash code 5460c190ef495baf43e2cd001687be272cd6a9d2, and all but one of the rest had the hash code d4b1d36c67149c981ea4b3d8392050188673817c. So if there's anything weird about this repository, it was thousands of tiny identical files in the same 'git annex assistant' adding pass. Once I scrubbed those files on later imports, things were considerably faster even when the file count increased dramatically.

You can go ahead and close this bug report if that seems reasonable—I just thought the weird behavior was worth reporting, in case somebody else runs it into again.

And thank you for an awesome program!

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.246.110"
 subject="comment 4"
 date="2013-07-26T22:28:37Z"
 content="""
Did some playing around, and I am seeing a progressive slowdown when adding lots of identical files in direct mode.
This does not happen when the files have different content, and thinking about it, it's pretty clear what's going on:

When there are a ton of files with the same content, the map file that has to list all the files using this content
get larger and larger. Each file added has to read and re-write the map file, which is obviously not going to scale well to thousands of duplicate files.
"""]]


code:
### Please describe the problem.

When running a command like `git annex copy --not --in bucket --to bucket`, I got:

`fatal: bad config file line 1 in /home/jim/tmp/git-annex14898.tmp`

I caught `git-annex14898.tmp` before it was deleted and it contained an HTML error page.
I have a remote `https://git.example.com/jim/annex.git`, and it appears that git-annex
is requesting `https://git.example.com/jim/annex.git/config`.  My server returns a 401
Forbidden and an error page for that URL, but git-annex tries to use the response as a config file anyway.

Jim

> [[fixed|done]] --[[Joey]]


code:
### Please describe the problem.

When attempting to clone and use a git repository in a subdirectory several levels deep on Windows, I observe symptoms very similar to those described at http://git-annex.branchable.com/direct_mode/#comment-8feee726df4e287dd3751bc77fd1441f.  By contrast, when I attempt the same operation in a subdirectory higher up, the operation is successful.  Logs of both sessions are given below.

My suspicion is that this has to do with exceeding the maximum path length limitation (MAX_PATH) of 260 characters on Windows, as described here: http://msdn.microsoft.com/en-us/library/aa365247.aspx.

### What steps will reproduce the problem?

See above.

### What version of git-annex are you using? On what operating system?

>git annex version
git-annex version: 5.20140517-gee56d21
build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier ddar hook external
local repository version: 5
supported repository version: 5
upgrade supported from repository versions: 2 3 4

>git version
git version 1.9.0.msysgit.0

Operating system: Windows 7 Professional (64-bit), Service Pack 1

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
C:\Users\areeves\Documents\Work\MyDirectoryHere\git>git clone ssh://areeves@myserver:/home/work/git/sbv
Cloning into 'sbv'...
remote: Counting objects: 65, done.
remote: Compressing objects: 100% (57/57), done.
remote: Total 65 (delta 26), reused 0 (delta 0)
Receiving objects: 100% (65/65), 9.25 KiB | 0 bytes/s, done.
Resolving deltas: 100% (26/26), done.
Checking connectivity... done.

C:\Users\areeves\Documents\Work\MyDirectoryHere\git>cd sbv
C:\Users\areeves\Documents\Work\MyDirectoryHere\git\sbv>git annex get

  Detected a filesystem without fifo support.

  Disabling ssh connection caching.

  Detected a crippled filesystem.

  Enabling direct mode.
git-annex: C:\Users\areeves\Documents\Work\MyDirectoryHere\git\sbv\.git\annex\objects\3de\5f4\SHA256-s765223180--c9e2eebd915b4ade9429b00a7a893df928389b3fb4ab759ea9f00b0e05e18de6\: openTempFile: does not exist (No such file or directory)

C:\Users\areeves\Documents\Work\MyDirectoryHere\git\sbv>git annex direct
commit
On branch master
Your branch is up-to-date with 'origin/master'.

nothing to commit, working directory clean
ok

git-annex: C:\Users\areeves\Documents\Work\MyDirectoryHere\git\sbv\.git\annex\objects\3de\5f4\SHA256-s765223180--c9e2eebd915b4ade9429b00a7a893df928389b3fb4ab759ea9f00b0e05e18de6\: openTempFile: does not exist (No such file or directory)
failed
git-annex: direct: 1 failed

C:\Users\areeves\Documents\Work\MyDirectoryHere\git\sbv>cd c:\temp
c:\temp>git clone ssh://areeves@myserver:/home/work/git/sbv
Cloning into 'sbv'...
remote: Counting objects: 65, done.
remote: Compressing objects: 100% (57/57), done.
remote: Total 65 (delta 26), reused 0 (delta 0)
Receiving objects: 100% (65/65), 9.25 KiB | 0 bytes/s, done.
Resolving deltas: 100% (26/26), done.
Checking connectivity... done.

c:\temp>cd sbv
c:\temp\sbv>git annex direct

  Detected a filesystem without fifo support.

  Disabling ssh connection caching.

  Detected a crippled filesystem.

  Enabling direct mode.
(Recording state in git...)

c:\temp\sbv>git annex get
get BigBinaryFile_Data_Package_2012-03-31.tar.bz2.gpg (merging origin/git-annex into git-annex...)
(Recording state in git...)
sent 30 bytes  received 765316741 bytes  11011752.10 bytes/sec
total size is 765223180  speedup is 1.00
ok
(Recording state in git...)

c:\temp\sbv>

# End of transcript or log.
"""]]


code:
The following commands show the failure:

$ mkdir d && touch d/f

$ mkdir g && cd g && git annex add ../d/f

add ... ok

error: Invalid path '.git/annex/objects/Jx/...

...

Then it seems it is enough to 'git add ../d/f' to complete the operation.

> Thanks for reporting, [[fixed|done]] --[[Joey]]


code:
### Please describe the problem.
Using assistant via xfce's "Internet -> Git annex" menu item, trying to connect to an ssh server fails

### What steps will reproduce the problem?
Use xubuntu 13.04 and git-annex

### What version of git-annex are you using? On what operating system?
git-annex version: 3.20121112ubuntu2

xubuntu 13.04

### Please provide any additional information below.
if I "sudo apt-get install ssh-askpass", this gets resolved. I think this should be fixed in the ubuntu repository's program dependencies.

Also if I run "git-annex webapp" from the terminal, this doesn't happen because it will ask me for the password in the terminal.
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
 Failed to ssh to the server. Transcript: ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory Permission denied, please try again. ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory Received disconnect from xxx.xxx.xxx.xxx: 2: Too many authentication failures for <user>

# End of transcript or log.
"""]]

> [[fixed|done]]; the assistant no longer recommends ssh-askpass, it does
> its own ssh password prompting and is its own ssh password agent.
> --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.255.110"
 subject="comment 1"
 date="2013-09-09T20:08:57Z"
 content="""
It's not a dependency. It's a recommends.

apt does not install new recommends that get added to a package. Adding that recommends only causes it to be installed by default when new users install the package

This seems to be the best I can do, short of adding a separate git-annex-assistant package, which could depend on ssh-askpass.
Might not be a bad idea..
"""]]


code:
### Please describe the problem.

Getting file from a linux rsync special remote from a windows machine is impossible.

### What steps will reproduce the problem?
git annex get .

### What version of git-annex are you using? On what operating system?
5.20140316-gbe47273 on windows 8.

### Please provide any additional information below.

Launching in debug mode, we see the wrong \\ used instead of / used. Syntax that rsync doesn't understand on a linux box.
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
read: rsync ["-e","'ssh' '-l' 'git' '-T'
","--progress","--inplace","git@strasser-family.ch:~/files/G5\\jj\\'SHA256E-s675
--1112f74fc9ebea01460efe26e447171a1aacf8e551f06f62ef6fe683df02e03c.mat\\SHA256E-
s675--1112f74fc9ebea01460efe26e447171a1aacf8e551f06f62ef6fe683df02e03c.mat'","/c
ygdrive/c/Users/pablo/Desktop/Nouveau dossier/InnerResult/.git/annex/tmp/SHA256E
-s675--1112f74fc9ebea01460efe26e447171a1aacf8e551f06f62ef6fe683df02e03c.mat"]

# End of transcript or log.
"""]]

> [[fixed|done]] --[[Joey]]


code:
### Please describe the problem.

When `git annex add` is run on dangling symlinks which point to the annex, the pre-commit hook `git annex pre-commit .` is run which, in turn, runs `git annex fix`.

During this fix, the mtime of the symlink changes. I assume the symlink is actually deleted and re-created in the background which would explain this behavior.

### What steps will reproduce the problem?

[[!format sh """
# manual import of annexed data into new annex
% mkdir repo1.annex
% cd !$
% git init; git annex init
% echo hi > foo
% git annex add foo; git commit -m bar
% mkdir ../repo2
% cd !$
% git init; git annex init
% mkdir .git/annex/objects/zQ/MQ/SHA256E-s3--98ea6e4f216f2fb4b69fff9b3a44842c38686ca685f3f55dc48c5d3fb1107be4/
% cp -ax ../repo1/.git/annex/objects/zQ/MQ/SHA256E-s3--98ea6e4f216f2fb4b69fff9b3a44842c38686ca685f3f55dc48c5d3fb1107be4/SHA256E-s3--98ea6e4f216f2fb4b69fff9b3a44842c38686ca685f3f55dc48c5d3fb1107be4 .git/annex/objects/zQ/MQ/SHA256E-s3--98ea6e4f216f2fb4b69fff9b3a44842c38686ca685f3f55dc48c5d3fb1107be4/SHA256E-s3--98ea6e4f216f2fb4b69fff9b3a44842c38686ca685f3f55dc48c5d3fb1107be4
% mkdir subdirectory
% cd !$
% cp -ax ../../repo1.annex/foo .
% ls -l foo
total 4
lrwxrwxrwx 1 richih richih 194 Jul 11 12:20 foo -> .git/annex/objects/zQ/MQ/SHA256E-s3--98ea6e4f216f2fb4b69fff9b3a44842c38686ca685f3f55dc48c5d3fb1107be4/SHA256E-s3--98ea6e4f216f2fb4b69fff9b3a44842c38686ca685f3f55dc48c5d3fb1107be4
% git annex fix # this does nothing
% ls -l foo
total 4
lrwxrwxrwx 1 richih richih 194 Jul 11 12:20 foo -> ..git/annex/objects/zQ/MQ/SHA256E-s3--98ea6e4f216f2fb4b69fff9b3a44842c38686ca685f3f55dc48c5d3fb1107be4/SHA256E-s3--98ea6e4f216f2fb4b69fff9b3a44842c38686ca685f3f55dc48c5d3fb1107be4
% git annex add foo
add foo ok
(Recording state in git...)
% ls -l foo
total 4
lrwxrwxrwx 1 richih richih 197 Jul 11 12:23 foo -> ../.git/annex/objects/zQ/MQ/SHA256E-s3--98ea6e4f216f2fb4b69fff9b3a44842c38686ca685f3f55dc48c5d3fb1107be4/SHA256E-s3--98ea6e4f216f2fb4b69fff9b3a44842c38686ca685f3f55dc48c5d3fb1107be4
%

"""]]

### What version of git-annex are you using? On what operating system?

git-annex 4.20130709 on Debian unstable i386 and x64.

### Please provide any additional information below.

As noted in [[todo/wishlist:___96__git_annex_import__96___--_An_easy_way_to_get_data_into_an_annex/#comment-e0ea68b5f84cf4130001ad34e9f3b2ff]], `git annex import` does not seem to work on other repos which is why I tried to build this by hand. This works fine and retains mtime as long as I don't need to change the relative depth the symlinks.
I think that the issue above should be fixed in and as of itself, though.

> [[fixed|done]] --[[Joey]]


code:
As per topic, `git annex forget --drop-dead --force` does not behave as expected. Instead of getting rid of dead repositories, it actually revives them.

I messed up the initial setup of the S3 special remote and instead of risking manual intervention, I figured it would be cleanest to nuke the old special remotes, using a new one instead.
And yes, I should have used a test repo :(

As this is apu.debconf.org, you could get access to the repo if that helps.

[[!format sh """
richih@apu (git)-[master] /srv/video/video.debian.net % git annex info
repository mode: indirect
trusted repositories: 0
semitrusted repositories: 4
	00000000-0000-0000-0000-000000000001 -- web
 	070cff8a-6302-4aa7-a63c-3fdd34e598a2 -- amazon_s3_us_east--SHA512E
 	0bae683f-bede-43dd-a815-c4f8fb6db32d -- aws_s3_us_east--SHA512E
 	92e9fac9-97ec-401f-a421-33f6b4f43e47 -- here (apu.debconf.org/srv/video/conference_videos.annex)
untrusted repositories: 0
transfers in progress: none
available local disk space: 136.3 gigabytes (+1 megabyte reserved)
local annex keys: 4392
local annex size: 884.64 gigabytes
annexed files in working tree: 4628
size of annexed files in working tree: 885.68 gigabytes
bloom filter size: 16 mebibytes (0.9% full)
backend usage:
	SHA512E: 9020
richih@apu (git)-[master] /srv/video/video.debian.net % git annex dead 070cff8a-6302-4aa7-a63c-3fdd34e598a2
dead 070cff8a-6302-4aa7-a63c-3fdd34e598a2 ok
(Recording state in git...)
richih@apu (git)-[master] /srv/video/video.debian.net % git annex dead 0bae683f-bede-43dd-a815-c4f8fb6db32d
dead 0bae683f-bede-43dd-a815-c4f8fb6db32d ok
(Recording state in git...)
richih@apu (git)-[master] /srv/video/video.debian.net % git annex initremote amazon_aws_s3-us_east_1--SHA512E type=S3 encryption='none' embedcreds='no' fileprefix='SHA512E/' bucket='debian-video'
initremote amazon_aws_s3-us_east_1--SHA512E (checking bucket...) git-annex: This bucket is already in use by a different S3 special remote, with UUID: 0bae683f-bede-43dd-a815-c4f8fb6db32d
richih@apu (git)-[master] /srv/video/video.debian.net % git annex info
repository mode: indirect
trusted repositories: 0
semitrusted repositories: 2
	00000000-0000-0000-0000-000000000001 -- web
 	92e9fac9-97ec-401f-a421-33f6b4f43e47 -- here (apu.debconf.org/srv/video/conference_videos.annex)
untrusted repositories: 0
transfers in progress: none
available local disk space: 136.3 gigabytes (+1 megabyte reserved)
local annex keys: 4392
local annex size: 884.64 gigabytes
annexed files in working tree: 4628
size of annexed files in working tree: 885.68 gigabytes
bloom filter size: 16 mebibytes (0.9% full)
backend usage:
	SHA512E: 9020
richih@apu (git)-[master] /srv/video/video.debian.net % git annex forget --drop-dead --force
forget git-annex (Recording state in git...)
ok
(Recording state in git...)
richih@apu (git)-[master] /srv/video/video.debian.net % git annex info
repository mode: indirect
trusted repositories: 0
semitrusted repositories: 4
	00000000-0000-0000-0000-000000000001 -- web
 	070cff8a-6302-4aa7-a63c-3fdd34e598a2 -- amazon_s3_us_east--SHA512E
 	0bae683f-bede-43dd-a815-c4f8fb6db32d -- aws_s3_us_east--SHA512E
 	92e9fac9-97ec-401f-a421-33f6b4f43e47 -- here (apu.debconf.org/srv/video/conference_videos.annex)
untrusted repositories: 0
transfers in progress: none
available local disk space: 136.3 gigabytes (+1 megabyte reserved)
local annex keys: 4392
local annex size: 884.64 gigabytes
annexed files in working tree: 4628
size of annexed files in working tree: 885.68 gigabytes
bloom filter size: 16 mebibytes (0.9% full)
backend usage:
	SHA512E: 9020
richih@apu (git)-[master] /srv/video/video.debian.net % git annex version
git-annex version: 5.20140117~bpo70+2
build flags: Assistant Webapp Pairing S3 Inotify DBus XMPP Feeds Quvi TDFA
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SHA256 SHA1 SHA512 SHA224 SHA384 WORM URL
remote types: git gcrypt S3 bup directory rsync web tahoe glacier hook external
local repository version: 5
supported repository version: 5
upgrade supported from repository versions: 0 1 2 4
richih@apu (git)-[master] /srv/video/video.debian.net % cat /etc/issue
Debian GNU/Linux 7 \n \l

richih@apu (git)-[master] /srv/video/video.debian.net %
"""]]

> [[fixed|done]] via not removing from trust.log --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.41"
 subject="comment 1"
 date="2014-03-26T17:19:58Z"
 content="""
You still have git remotes configured with the uuid of the remote, so `git annex info` pulls that data in. Since --drop-dead removes all mention of the remote from the git-annex branch, including that its trust level is dead, the remaining info from the .git/config takes effect.

--drop-dead could `git remote rm` but that won't help other clones of the repo that also have the dead remote. Instead, I think it may make sense for --drop-dead to avoid removing the uuid from trust.log, so it will still know this remote is dead.

(Of course, you can easily deal with this locally by `git remote rm` yourself.)
"""]]