code:
### Please describe the problem.
The webapp does not offer me the option to delete a remote repository that it did not succeed to synchronize with. Status of the repository is sync enabled (metadata only) but in settings I may only edit the repo, but cannot delete it.

### What steps will reproduce the problem?
Set up git-annex on a new computer. Forget to enable ssh on the machine. Perform a local pairing (success). The sync fails for obvious reasons.

### What version of git-annex are you using? On what operating system?
System is ubuntu but I assume this is a general issue. Version of git-annex is 5.20140117


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.46"
 subject="comment 1"
 date="2014-01-26T17:57:25Z"
 content="""
If you go to Edit, it will let you upgrade the repository, which should fix your problem.
"""]]

repo_name: kakafarm/git

license: unknown


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmXtqLEfN6cGcF9gW_09QUDoRpsexSdRcM"
 nickname="Thomas"
 subject="comment 2"
 date="2014-01-26T18:12:08Z"
 content="""
Unfortunately, upgrading did not fix it. I suppose, a restart of the remote server (ssh was just installed) would have been necessary. Instead, I just deleted the repository on the remote.

Now my local repository was caught in a situation, where I couldn't upgrade the broken remote and I couldn't delete it either.

By recreating the repository on the remote, I finally was able to get rid of the broken link, but I think the proper solution would be to have the ability to remove unsynced repositories.

Thanks for the quick response!
"""]]


code:
### Please describe the problem.

Any file containing non ascii chars either won't get commited by the webapp or when clone of that repo is made they are shown as deleted in git status but the files are there.

you can not commit them neither,

    git commit -m 'foo'

or

    git annex sync

will get them to commit trying to reset the state using,

    git checkout master

or

    git clean -f

won't fix the issue. till the issue is fixed sync does not work from the command line since there are changes. Temporarily running assistant in the directory does fix the problem but every now and then it pops again.

Also moving any symlinks around containing non ascii chars also causes the same issue. git annex won't add the moved symlinks git annex fix or git annex add won't add them git annex sync also skips them again the only solution is to either run the assistant or restart it if it is running.

### What steps will reproduce the problem?

Adding any file with a non ascii char on OS X (10.9).

### What version of git-annex are you using? On what operating system?

git-annex version: 5.20140421-gc34a665

### Please provide any additional information below.

Also another thing I noticed is that same file names same repo will clone fine on Linux no errors this only happens on OS X.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

# End of transcript or log.
"""]]

[[!tag moreinfo]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.36"
 subject="comment 1"
 date="2014-05-24T19:23:08Z"
 content="""
I think you are going to need to provide more information:

* Some actual file names that have the problem. \"non-ascii chars\" is a very wide description, you don't even say if you're using unicode or what.
* Some error messages or transcripts of what happens when you see the problem.
* What is your locale setting on OSX.

I tried with some unicode filenames on OSX and they were committed without difficulty in the en_US.utf8 locale.
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/FHnTlSBo1eCGJRwueeKeB6.RCaPbGMPr5jxx8A--#ce0d8"
 nickname="Hamza"
 subject="comment 2"
 date="2014-05-25T15:16:33Z"
 content="""
I file name I remember messing is the following,

    George's Café

I experience this problem with any char that is not ascii. As for my locale,

LANG=\"en_US.UTF-8\"
LC_COLLATE=\"en_US.UTF-8\"
LC_CTYPE=\"en_US.UTF-8\"
LC_MESSAGES=\"en_US.UTF-8\"
LC_MONETARY=\"en_US.UTF-8\"
LC_NUMERIC=\"en_US.UTF-8\"
LC_TIME=\"en_US.UTF-8\"
LC_ALL=\"en_US.UTF-8\"

There no errors other than the behaviour I've described.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.176"
 subject="comment 3"
 date="2014-05-27T18:02:11Z"
 content="""
I tried with that filename, and had no difficulty committing it.

<pre>
oberon:repo joeyh$ git annex add George\'s\ Cafe\314\201
add George's Café ok
(Recording state in git...)
oberon:repo joeyh$ git commit -m foo
[master (root-commit) 8acd5d4] foo
 Committer: Joey Hess <joeyh@oberon.local.tam-lin.net>
Your name and email address were configured automatically based
on your username and hostname. Please check that they are accurate.
You can suppress this message by setting them explicitly:

    git config --global user.name \"Your Name\"
    git config --global user.email you@example.com

After doing this, you may fix the identity used for this commit with:

    git commit --amend --reset-author

 1 file changed, 1 insertion(+)
 create mode 120000 \"George's Caf\303\251\"

</pre>

I even tried running the assistant with all locale settings set to C, and it still had no problem adding and committing that filename.

So, I need help reproducing this, or I need to see the error message you see when the problem happens to you.
"""]]


code:
### Please describe the problem.
[[!format sh """
cabal install -O2 -j1 -f-webdav -f-s3 git-annex
Resolving dependencies...
Configuring dns-1.2.0...
Building dns-1.2.0...
Preprocessing library dns-1.2.0...

Network/DNS/Decode.hs:15:8:
    Could not find module `Data.Conduit.Network'
    It is a member of the hidden package `conduit-extra-1.1.0'.
    Perhaps you need to add `conduit-extra' to the build-depends in your .cabal file.
    Use -v to see a list of the files searched for.
Failed to install dns-1.2.0
cabal: Error: some packages failed to install:
dns-1.2.0 failed during the building phase. The exception was:
ExitFailure 1
git-annex-5.20140402 depends on dns-1.2.0 which failed to install.
"""]]

### What steps will reproduce the problem?

[[!format sh """
cabal update
mkdir -p ~/haskell/git-annex
cd ~/haskell/git-annex
cabal sandbox init
cabal install -O2 -j1 -f-webdav -f-s3 c2hs git-annex
"""]]
### What version of git-annex are you using? On what operating system?

5.20140402, Gentoo Linux

### Please provide any additional information below.

> [[fixed|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl1D_4vD5ueaDw8gRsIYPO3UHRKEpFfg9I"
 nickname="Dmitry"
 subject="comment 1"
 date="2014-04-04T06:40:21Z"
 content="""
Author of \"dns\" library already fixed this issue.

Next error is:
[[!format sh \"\"\"
Preprocessing executable 'git-annex' for git-annex-5.20140402...

Utility/Yesod.hs:36:8:
    Could not find module `Text.Hamlet'
    It is a member of the hidden package `shakespeare-2.0.0.1'.
    Perhaps you need to add `shakespeare' to the build-depends in your .cabal file.
    Use -v to see a list of the files searched for.
cabal: Error: some packages failed to install:
\"\"\"]]

Here is the fix:

[[!format diff \"\"\"
Index: git-annex/git-annex.cabal
===================================================================
--- git-annex.orig/git-annex.cabal
+++ git-annex/git-annex.cabal
@@ -101,7 +101,7 @@ Executable git-annex
    base (>= 4.5 && < 4.9), monad-control, MonadCatchIO-transformers,
    IfElse, text, QuickCheck >= 2.1, bloomfilter, edit-distance, process,
    SafeSemaphore, uuid, random, dlist, unix-compat, async, stm (>= 2.3),
-   data-default, case-insensitive
+   data-default, case-insensitive, shakespeare
   CC-Options: -Wall
   GHC-Options: -Wall
   Extensions: PackageImports
\"\"\"]]
"""]]


code:
### Please describe the problem.

On my raspberry pi, an SSH remote was in /var/lib/store (symlinked with /home/carlo/store), which contained a LVM volume spanning several USB drives. One of the drives got corrupted, which lead to this:

$ cd /home/carlo/store
$ ls
Input/Output Error

On my desktop machine, I then had a lot of transfers. After transfer, the file was turned into a symlink.

### What steps will reproduce the problem?

* Create an annex and a remote ssh server.
* [Simulate a corrupted drive](http://stackoverflow.com/questions/1361518/how-can-i-simulate-a-failed-disk-during-testing) for the remote, creating an input output error.
* Wait until the annex starts syncing on the desktop machine. If there were no further unidentified causes on my side, the assistant will start transfers that revert files to symlinks.

### What version of git-annex are you using? On what operating system?

Ubuntu 12.04 64bit
git annex 3.20121017 ([from Ubuntu PPA](https://launchpad.net/~rubiojr/+archive/git-annex))

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

Please see [all my daemon.log](http://capocasa.name/daemon.tgz) files.

I noticed the problem yesterday afternoon (Thu 24 Oct).

# End of transcript or log.
"""]]

> [[!taglink moreinfo]]; either I don't have enough information to work on this,
> or it might have just been user error. --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 1"
 date="2013-10-26T19:22:57Z"
 content="""
It seems to me that if a subdirectory of the repository is on a corrupted drive, and so it's not possible to list the files on it, this is basically the same as if you'd 'rm -rf' that subdirectory. So when it starts up, the assistant will see that these files are not present, and commit a removal to git.

Then when another machine syncs with that, it would delete the files from its repository too. However, it actually keeps the contents of the files stashed away in `.git/annex`. So to recover from this, all you have to do is `git annex indirect` and `git revert` the commit that deleted the files. All your files would then be available again.

However, what you describe is instead that the assistant chose to drop the content associated with the files, but kept the symlinks for them checked into git.
I don't understand why it would do that. Can you show the output of running, on the desktop machine:

    git annex whereis $somefile
    git annex get $somefile

Where $somefile is one of the files that has been reduced to a symlink.

Looking at your logs, they appear to be the logs from the server. The strange thing that appears in one of them is \"git-annex: Not in a git repository.\"
which was logged around 2013-10-24 20:07:25 CEST. I am not sure, but I think it might have been the rpi git-annex saying that, because there is also \"fatal: '~/store/annex/' does not appear to be a git repository\"
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnZEanlyzay_QlEAL0CWpyZcRTyN7vay8U"
 nickname="Carlo"
 subject="comment 2"
 date="2013-10-27T19:26:42Z"
 content="""
Thanks for looking into it and the recover instructions, will recover and post back when I'm back at work. I also have an another indirect mode repo on my home desktop as an extra backup so nothing real bad happened, but it's reassuring data's still there at work.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnZEanlyzay_QlEAL0CWpyZcRTyN7vay8U"
 nickname="Carlo"
 subject="comment 5"
 date="2013-10-30T15:05:37Z"
 content="""
The assistant autorecovered my work repo before I noticed, so it looks like I can't provide the necessary info. There were a bunch of files missing that got re-synced from my home PC.

For what it's worth, I noticed that on my phone, when cutting the internet connection while syncing, the assistant downloaded existing files into placeholder files, and then continued actually downloading files when the network connection was restored.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 5"
 date="2013-11-03T01:17:39Z"
 content="""
I don't understand what you mean by \"The assistant autorecovered my work repo before I noticed\". What repo is the work repo, and how could the assistant \"autorecover\" it, and what did it do?

At this point, I am completely in the dark about whether you're reporting a problem, and what the problem is.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnZEanlyzay_QlEAL0CWpyZcRTyN7vay8U"
 nickname="Carlo"
 subject="comment 6"
 date="2013-10-31T10:03:01Z"
 content="""
Of course, the autorecovery wouldn't affect the daemon.log from the work machine... so here they are: http://capocasa.name/work-desktop-ubuntu-12.04-daemonlog.tgz
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnZEanlyzay_QlEAL0CWpyZcRTyN7vay8U"
 nickname="Carlo"
 subject="comment 6"
 date="2013-11-19T09:48:26Z"
 content="""
Sorry, missed the comment.

My work repo is the repository on my work laptop, where deletions got synced to.

Git annex had then run repository repair automatically, so the odd symlinks where no longer there for me to check out.

It is possible that I ran some git commands in direct mode I shouldn't have; I put the files back in and it's working nicely now. So this might have been a \"no direct mode guard\" issue.

"""]]


code:
### Please describe the problem.

I've been using git-annex for a few weeks now, and everything was working fine until today. Now, when git-annex goes to sync my files, it will hang for a while, and then display a red box on the left side, saying `! Synced with adam.liter`. So, basically, it doesn't really sync. The logs say that it cannot read from the remote repository, which is set up on box.com and was working just fine until today.

I've tried reconfiguring my jabber account on all my devices with git-annex as well as deleting and remaking the box.com repository, and none of these steps have solved the problem.

### What steps will reproduce the problem?

Making changes to any file located in the repository.

### What version of git-annex are you using? On what operating system?

Mac OS X 10.8.4 and git-annex version 4.20130801-gc88bbc4.

### Please provide any additional information below.

[[!format sh """
[2013-09-09 01:33:03 EDT] XMPPSendPack: Syncing with adam.liter
Already up-to-date.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
"""]]

[[!meta title="xmpp syncing sometimes fails"]]

[[!tag confirmed]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.255.110"
 subject="comment 1"
 date="2013-09-09T06:01:18Z"
 content="""
From the small bit of log posted, it seems to be failing to sync over XMPP. Perhaps the XMPP server is down or it cannot connect? You can turn on debugging to get a lot more debugging about XMPP sync attempts.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkgH7oNEqNbh3g-N1-UHXuqleXaRYDgj1U"
 nickname="Adam"
 subject="comment 2"
 date="2013-09-09T06:12:44Z"
 content="""
This is what is in the logs after I turned on debugging:

    [2013-09-09 02:04:22 EDT] XMPPSendPack: finished running push Pushing \"a29\" (PushRequest (UUID \"c96c8493-b702-4ca0-9a88-f8a38dd6c3f5\")) True
    [2013-09-09 02:04:22 EDT] XMPPSendPack: started running push Pushing \"a29\" (PushRequest (UUID \"c96c8493-b702-4ca0-9a88-f8a38dd6c3f5\"))
    [2013-09-09 02:04:22 EDT] read: git [\"--git-dir=/Users/adamliter/Library/texmf/.git\",\"--work-tree=/Users/adamliter/Library/texmf\",\"symbolic-ref\",\"HEAD\"]
    [2013-09-09 02:04:22 EDT] read: git [\"--git-dir=/Users/adamliter/Library/texmf/.git\",\"--work-tree=/Users/adamliter/Library/texmf\",\"show-ref\",\"refs/heads/master\"]
    [2013-09-09 02:04:22 EDT] call: git [\"--git-dir=/Users/adamliter/Library/texmf/.git\",\"--work-tree=/Users/adamliter/Library/texmf\",\"branch\",\"-f\",\"synced/master\"]
    [2013-09-09 02:04:22 EDT] XMPPSendPack: Syncing with adam.liter
    [2013-09-09 02:04:22 EDT] call: git [\"--git-dir=/Users/adamliter/Library/texmf/.git\",\"--work-tree=/Users/adamliter/Library/texmf\",\"push\",\"adam.liter\",\"git-annex:refs/synced/6258750b-9b0d-4f05-b443-61f4fb0a3f66/YWRhbS5saXRlckBnbWFpbC5jb20=/git-annex\",\"refs/heads/master:refs/synced/6258750b-9b0d-4f05-b443-61f4fb0a3f66/YWRhbS5saXRlckBnbWFpbC5jb20=/master\"]
    [2013-09-09 02:04:22 EDT] XMPPClient: sending: Pushing \"a29\" (StartingPush (UUID \"6258750b-9b0d-4f05-b443-61f4fb0a3f66\"))
    [2013-09-09 02:04:22 EDT] XMPPClient: to client: a10/F4E99F98
    [2013-09-09 02:04:23 EDT] XMPPReceivePack: timeout waiting for git send-pack output via XMPP
    fatal: The remote end hung up unexpectedly
    [2013-09-09 02:04:23 EDT] XMPPReceivePack: finished running push Pushing \"a29\" (StartingPush (UUID \"c96c8493-b702-4ca0-9a88-f8a38dd6c3f5\")) True
    [2013-09-09 02:04:23 EDT] XMPPClient: sending: Pushing \"a29\" (ReceivePackDone (ExitFailure 128))
    [2013-09-09 02:04:23 EDT] XMPPReceivePack: started running push Pushing \"a29\" (StartingPush (UUID \"c96c8493-b702-4ca0-9a88-f8a38dd6c3f5\"))
    [2013-09-09 02:04:23 EDT] XMPPClient: to client: a10/F4E99F98
    [2013-09-09 02:04:23 EDT] XMPPReceivePack: Syncing with adam.liter
    [2013-09-09 02:04:23 EDT] chat: git [\"receive-pack\",\"/Users/adamliter/Library/texmf\"]
    [2013-09-09 02:04:23 EDT] XMPPClient: sending: Pushing \"a29\" (ReceivePackOutput 1 \"<elided>\")
    [2013-09-09 02:04:23 EDT] XMPPClient: to client: a10/F4E99F98
    [2013-09-09 02:04:23 EDT] XMPPClient: sending: Pushing \"a29\" (ReceivePackOutput 2 \"<elided>\")
    [2013-09-09 02:04:23 EDT] XMPPClient: to client: a10/F4E99F98
    [2013-09-09 02:04:23 EDT] XMPPClient: received: [\"Pushing \\"a29\\" (StartingPush (UUID \\"c96c8493-b702-4ca0-9a88-f8a38dd6c3f5\\"))\"]
    [2013-09-09 02:04:23 EDT] XMPPClient: received: [\"Unknown message\"]

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnvVfFLW4CTKs7UjdiLIsOn_cxj1Jnh64I"
 nickname="Charl"
 subject="Could it be gmail.com XMPP throttling?"
 date="2014-03-23T19:26:08Z"
 content="""
I was seeing similar error messages, until I registered for a different XMPP account at jabber.de and started using that instead of my gmail.com account.

My current suspicion is that it could be Google performing throttling on their XMPP service. See here: http://stackoverflow.com/questions/1843837/what-is-the-throttling-rate-that-gtalk-applies-to-xmpp-messages

"""]]


code:
### Please describe the problem.
The watcher crashes.

I only need to restart the thread in the pop-up to get everything to work again, but I'm reporting just in case that this issue has any other implications.

### What steps will reproduce the problem?
I open the webapp and in the minutes before it starts syncing (syncing is enabled) I disable it (clicking in the 'syncing enabled' text).

This produces a crash every time.

### What version of git-annex are you using? On what operating system?
5.20140320 in Debian sid and testing

### Please provide any additional information below.
This is all I can see in the logs

[[!format sh """
Watcher crashed: PauseWatcher
[2014-03-26 08:54:57 CET] Watcher: warning Watcher crashed: PauseWatcher
"""]]

[[!tag moreinfo]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.41"
 subject="comment 1"
 date="2014-03-26T17:36:00Z"
 content="""
How did you install git-annex? Is this Debian Linux?

I have not been able to reproduce a crash. It's indeed the case that a PauseWatcher exception is thrown, but the Watcher explicitly catches that exception.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawk9nck8WX8-ADF3Fdh5vFo4Qrw1I_bJcR8"
 nickname="Jon Ander"
 subject="comment 2"
 date="2014-04-01T08:04:51Z"
 content="""
Yes, this is Debian Linux and I've been able to reproduce it in i386 and amd64. git-annex is installed from the Debian repositories.

I'll try to continue testing the issue and will report back if I can find any useful info.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.244"
 subject="comment 3"
 date="2014-04-02T20:38:12Z"
 content="""
So we have the same version of git-annex from the same build, and only you see the problem. Hmm..

You mentioned that you see the problem if you disable syncing at a particular time. Does it only crash at that time, or at any time?

If you create a brand new empty repository and run the webapp in it, can you reporoduce the problem there? Ie, \"mkdir test; cd test; git init; git annex init; git annex webapp\"
"""]]


code:
### Please describe the problem.
When creating a https webdav repository under openbsd it complains that /etc/ssl/certs doesn't exist. This is true considering all certs are stored in /etc/ssl/cert.pem.
After /etc/ssl/certs is created it complains about that the certificate has an unknown CA, for obvious reasons.

A workaround is to symlink /etc/ssl/cert.pem in /etc/ssl/certs

### What steps will reproduce the problem?
See below

### What version of git-annex are you using? On what operating system?
5.20140129 under OpenBSD 5.4

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
WEBDAV_USERNAME=<username> WEBDAV_PASSWORD=<password> git annex initremote box.com type=webdav url=https://dav.box.com/dav/Documents chunksize=100mb encryption=hybrid keyid=<key> mac=HMACSHA512
initremote box.com (encryption setup) (hybrid cipher with gpg key <key>) (testing WebDAV server...)

git-annex: WebDAV failed to write file: /etc/ssl/certs/: getDirectoryContents: does not exist (No such file or directory): user error
failed
git-annex: initremote: 1 failed

# End of transcript or log.
"""]]

> This needs to be fixed in the haskell certificate library.
> I have filed a bug there:
> <https://github.com/vincenthz/hs-certificate/issues/26>
>
> Patch would probably be pretty simple. Based on description, something like
> this:

[[!format patch """
diff --git a/System/Certificate/X509/Unix.hs b/System/Certificate/X509/Unix.hs
index 8463465..74316e9 100644
--- a/System/Certificate/X509/Unix.hs
+++ b/System/Certificate/X509/Unix.hs
@@ -50,7 +50,7 @@ listDirectoryCerts path = (map (path </>) . filter isCert <$> getDirectoryConten
           isCert x = (not $ isPrefixOf "." x) && (not $ isHashedFile x)

 getSystemCertificateStore :: IO CertificateStore
-getSystemCertificateStore = makeCertificateStore . concat <$> (getSystemPath >>= listDirectoryCerts >>= mapM readCertificates)
+getSystemCertificateStore = makeCertificateStore <$> readCertificates "/etc/ssl/cert.pem"

 getSystemPath :: IO FilePath
 getSystemPath = E.catch (getEnv envPathOverride) inDefault
"""]]

>
> [[closing|done]] as no changes to git-annex can fix this. --[[Joey]]


code:
### Please describe the problem.

When creating a new remote with the webapp (on Windows) and setting the remote repository directory as "~/wherever", everything seems to be going well up until pressing the button to create an unencrypted repository. There is no output from either the terminal or error messages displayed in the browser.

On the remote server, it turns out that it is creating a directory named ~ within $HOME so that there is a $HOME/~/wherever directory.

Omitting the leading ~/ correctly creates $HOME/wherever and the setup proceeds as it should.

### What steps will reproduce the problem?

Try to create a remote repository with directory "~/wherever".

### What version of git-annex are you using? On what operating system?

OS: Windows 7

Output of git annex version:

git-annex version: 5.20140517-gee56d21
build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV DNS Feed
s Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SH
A256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier ddar ho
ok external

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

# End of transcript or log.
"""]]

> [[fixed|done]] --[[Joey]]


code:
What steps will reproduce the problem?

Don't have gpg installed/on your $PATH, and attempt to create an encrypted S3 remote via the web interface.

What is the expected output? What do you see instead?

Expected to be told to install GPG. Actual output was a Yesod error:

Internal Server Error
user error (gpg ["--batch","--no-tty","--use-agent","--quiet","--trust-model","always","--gen-random","--armor","1","512"] exited 127)

What version of git-annex are you using? On what operating system?

3.20130107 on Fedora 17 (64-bit).

Please provide any additional information below.

[[!tag /design/assistant]]
[[!tag confirmed]]


code:
### Please describe the problem.
Using the git-annex webapp in remote mode will forward to the wrong IP (localhost) when creating the second repository

### What steps will reproduce the problem?
Needs two computers (C1, C2)

- C1: run "git-annex webapp --listen=[IP of C1]
- C2: use a browser to go to the address you just got on C1 (should contain IP of C1)
- C2: create an repository as prompted (should work fine)
- C2: In the dropdown on the top right select "Add another repository", choose desired location, select keep separate
- C2: Browser forwards to new address, but instead of the IP of C1, 127.0.0.1 is used (which obviously fails)

### What version of git-annex are you using? On what operating system?

[[!format sh """
$ git-annex version
git-annex version: 4.20130802
build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP
$ lsb_release -d
Description:    Ubuntu 13.04
$ uname -a
Linux nas 3.8.0-27-generic #40-Ubuntu SMP Tue Jul 9 00:19:35 UTC 2013 i686 i686 i686 GNU/Linux
"""]]

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

# End of transcript or log.
"""]]

> This is a different effect of the same bug in [[Hangs on creating repository when using --listen]]. Closing as [[dup|done]] --[[Joey]]


code:
### Please describe the problem.
If have a directory - mounted CIF share (windows server with domain).
If i run "git annex assistant" or "git annex watch" daemon starts fine and works at first, but then after some time stops doing anything: no messages in .git/annex/daemon.log, process hangs in ps aux.

If i try to "cat .git/annex/daemon.pid" I get "permission denied". And permission is denied for everyone: root, from other windows machines, etc.

If i then kill daemon process and start daemon again everything repeats.

### What version of git-annex are you using? On what operating system?
Standalone git-annex 5.20140421-g515d251 on CentOS 6.5

> [[done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmaB2n9HFclAvyMyDzvTPaifh5OrlF51Co"
 nickname="VS7"
 subject="comment 1"
 date="2014-05-21T07:17:44Z"
 content="""
[[!format sh \"\"\"
[filesync@ServiceVM00001 manuals]$ ps aux | grep assistant
filesync 27036  0.0  0.0 103244   864 pts/0    S+   11:13   0:00 grep assistant
\"\"\"]]
[[!format sh \"\"\"
[filesync@ServiceVM00001 manuals]$ git annex assistant
\"\"\"]]
[[!format sh \"\"\"
[filesync@ServiceVM00001 manuals]$ cat .git/annex/daemon.log
[2014-05-21 11:14:06 MSK] main: starting assistant version 5.20140421-g515d251
[2014-05-21 11:14:07 MSK] Cronner: You should enable consistency checking to protect your data.
failed to add watch on directory /home/filesync/.config/git-annex (/home/filesync/.config/git-annex: getDirectoryContents: does not exist (No such file or directory))

  dbus failed; falling back to mtab polling (ClientError {clientErrorMessage = \"runClient: unable to determine DBUS address\", clientErrorFatal = True})
p11-kit: couldn't load module: /usr/lib/x86_64-linux-gnu/pkcs11/p11-kit-trust.so: /usr/lib/x86_64-linux-gnu/pkcs11/p11-kit-trust.so: cannot open shared object file: No such file or directory

  No known network monitor available through dbus; falling back to polling
[2014-05-21 11:14:09 MSK] Upgrader: An upgrade of git-annex is available.  (version 5.20140517)

Already up-to-date.
(scanning...) [2014-05-21 11:14:09 MSK] Watcher: Performing startup scan
\"\"\"]]
[[!format sh \"\"\"
[filesync@ServiceVM00001 manuals]$ git annex assistant --stop

git-annex: /home/filesync/manuals/.git/annex/daemon.pid: hGetContents: permission denied (Permission denied)
failed
git-annex: assistant: 1 failed
\"\"\"]]
[[!format sh \"\"\"
[filesync@ServiceVM00001 manuals]$ cat .git/annex/daemon.pid
cat: .git/annex/daemon.pid: Permission denied
\"\"\"]]
[[!format sh \"\"\"
[filesync@ServiceVM00001 manuals]$ ps aux | grep assistant
filesync 27060  6.2  0.5 566372 41808 ?        Sl   11:14   0:04 /home/filesync/git-annex.linux//lib64/ld-linux-x86-64.so.2 --library-path /home/filesync/git-annex.linux//usr/lib/x86_64-linux-gnu/gconv:/home/filesync/git-annex.linux//usr/lib/x86_64-linux-gnu/audit:/home/filesync/git-annex.linux//etc/ld.so.conf.d:/home/filesync/git-annex.linux//usr/lib:/home/filesync/git-annex.linux//usr/lib/x86_64-linux-gnu:/home/filesync/git-annex.linux//lib64:/home/filesync/git-annex.linux//lib/x86_64-linux-gnu: /home/filesync/git-annex.linux/shimmed/git-annex/git-annex assistant
filesync 27115  0.0  0.0 103244   864 pts/0    S+   11:15   0:00 grep assistant
\"\"\"]]
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.3"
 subject="comment 2"
 date="2014-05-21T17:11:27Z"
 content="""
It seems to me that your filesystem is broken.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmaB2n9HFclAvyMyDzvTPaifh5OrlF51Co"
 nickname="VS7"
 subject="comment 3"
 date="2014-05-22T07:23:23Z"
 content="""
*1. Windows Server 2012, deduplication is on.*
[[!format sh \"\"\"
[filesync@ServiceVM00001 manuals]$ ps aux | grep assist
filesync  1222  0.0  0.0 103244   860 pts/0    S+   11:20   0:00 grep assist
[filesync@ServiceVM00001 manuals]$ git annex assistant
[filesync@ServiceVM00001 manuals]$ cat .git/annex/daemon.pid
1250[filesync@ServiceVM00001 manuals]$ git annex assistant

git-annex: /home/filesync/manuals/.git/annex/daemon.pid: hGetContents: permission denied (Permission denied)
failed
git-annex: assistant: 1 failed
[filesync@ServiceVM00001 manuals]$ cat .git/annex/daemon.pid
cat: .git/annex/daemon.pid: Permission denied
\"\"\"]]

*2. Windows 7 Pro, no deduplication.*
[[!format sh \"\"\"
[filesync@ServiceVM00001 test]$ ps aux | grep assistant
filesync   655  0.0  0.0 103244   856 pts/0    S+   11:18   0:00 grep assistant
[filesync@ServiceVM00001 test]$ git annex assistant
[filesync@ServiceVM00001 test]$ cat .git/annex/daemon.pid
cat: .git/annex/daemon.pid: Permission denied
[filesync@ServiceVM00001 test]$ git annex assistant

git-annex: /home/filesync/test/.git/annex/daemon.pid: hGetContents: permission denied (Permission denied)
failed
git-annex: assistant: 1 failed
\"\"\"]]

*3. Local directory. Normal behavior*
[[!format sh \"\"\"
[filesync@ServiceVM00001 test1]$ ps aux | grep assist
filesync  1315  0.0  0.0 103244   856 pts/0    R+   11:23   0:00 grep assist
[filesync@ServiceVM00001 test1]$ git annex assistant
[filesync@ServiceVM00001 test1]$ cat .git/annex/daemon.pid
1338[filesync@ServiceVM00001 test1]$ git annex assistant
git-annex: Daemon is already running.
[filesync@ServiceVM00001 test1]$ cat .git/annex/daemon.pid
1338
\"\"\"]]
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmaB2n9HFclAvyMyDzvTPaifh5OrlF51Co"
 nickname="VS7"
 subject="comment 4"
 date="2014-05-22T09:44:33Z"
 content="""
Now i begin to suspect that daemon doesn't work with CIF mounted directories at all.

    mkdir ~/test-container
    mkdir ~/test-container/test
    mount -t cifs -o user=test_user,uid=502,gid=502 //srv/test ~/test-container/test
    cd ~/test-container
    git init
    git annex init
    git annex direct
    git annex add
    git annex sync
    git annex assistant

Then if i place any file in ~/test-container then git-annex sees changes and processes it almost immediately. But if i do anything inside mounted ~/test-container/test daemon ignores it.
\"git annex status\" says that yes there are changes inside \"~/test-container/test\" and i can commit them via \"git annex add\" & \"git annex sync\", but daemon ignores.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="24.159.78.125"
 subject="comment 5"
 date="2014-05-23T14:34:34Z"
 content="""
My point is that if the filesystem is rejecting all attempts to access it with \"permission denied\", it's hardly surprising that git-annex doesn't work, since nothing works. If running the assistant somehow causes the filesystem to get into this strange broken state, then you have a nice repoducible bug to report on the filesystem, I suppose. I don't see how it could be a bug in git-annex, since it is using only standard posix filesystem system calls, plus inotify.

Based on this, samba/CIFS does not support inotify: <http://stackoverflow.com/questions/8124617/getting-file-create-notifications-for-cifs-mount-in-linux>
Thus, the assistant won't work on it.

If I were you, I'd set up git-annex repositories on local filesystems, and let it take care of the syncronosation, rather than using a networked filesystem.
"""]]


code:
### Please describe the problem.
I created a remote repo on ssh server with the same git-annex version, no personal ssh keys for the repo (password authentication). But I put ~ in front of the repo name so it created in different place than I wanted. I deleted it from assistant and then deleted the remote repo dir. When I added it with the correct path again it always asks for server password (it has some old annex pub key in authorized_keys, but not the new one; it might have been prompted for the password also with the initial repo). During the whole thing it failed to connect a few times due to wrong password or me realizing too late that it asks for yet another one and it generated keys few times, but ultimately didn't put the last one in remote's authorized_keys.

### What steps will reproduce the problem?
Create, delete and create again a new repo on remote ssh server with password auth.

### What version of git-annex are you using? On what operating system?
git-annex 4.20130601 Gentoo amd64

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

# End of transcript or log.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.193"
 subject="comment 1"
 date="2013-06-25T18:20:05Z"
 content="""
It's true that deleting a ssh repository in the webapp does not delete any ssh keys that the webapp configured to access that repository. I'm a bit reluctant to try to clean it up entirely automatically, because the ssh configuration might have been manually altered, or might be used by another local repository. Particularly, if another local repo is set up to use the same ssh remote, the webapp will reuse the ssh key.

> Create, delete and create again a new repo on remote ssh server with password auth.

I tried doing this, and did not encounter any problem. Since the new repo had a different path on the ssh server, I simply ended up with two ssh keys configured in `~/.ssh/config` for two different repositories. The presence or absence of the first key did not affect the second key.

I think you need to go into more detail about exactly what you did, or show the files in .ssh that were set up, so I can see what the actual problem is.
"""]]


code:
### Please describe the problem.

When a repository is set in direct mode it will still replace files with symlinks when it becomes aware of a change but still hasn't been able to sync the file contents. This can create repositories that are temporarily unusable with files replaced with broken symlinks.

### What steps will reproduce the problem?

1. Create two repositories with each other as remotes
2. Run the assistant on both
3. Create some file changes in one and watch the directory in another.
4. For a brief (or sometimes long) time the destination repository will have it's old version of the file replaced by a broken symlink

This is particularly noticeable when using XMPP as it can often be the case that the two repositories can't connect to each other directly but can talk through XMPP. This breaks using git-annex in direct mode for things like having a synced config directory across machines. Something like having "~/.bashrc" linked into "~/annex-repository/bashrc", doesn't work as there will be times when a machine is broken because .bashrc is linked to a broken symlink while it fetches a new version.

The desired behavior would be to have git-annex in direct mode only replace older versions of files with newer versions of files.

### What version of git-annex are you using? On what operating system?

[[!format sh """
$ git annex version
git-annex version: 4.20130516.1
build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP
local repository version: 4
default repository version: 3
supported repository versions: 3 4
upgrade supported from repository versions: 0 1 2
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 12.04.2 LTS
Release:	12.04
Codename:	precise
"""]]

[[!meta title="git-annex does not guarantee modified files are available when merging with a new version of a repository"]]

[[!tag confirmed]]


code:
I did a "git annex add" of a bunch of files on a storage server with low IOPS, and saw this:

    git-annex: /tank/Media/Pictures/.git/annex/tmp/430_32b_SHA256E-s4464838--c1785a76ee1451f602e93c99c147e214705004e541de8256d74a3be3717d15e5.jpg.log: openBinaryFile: resource busy (file is locked)
failed

How is that even possible, when the server is doing nothing else?

[[!tag moreinfo]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.211"
 subject="comment 1"
 date="2013-01-05T21:13:09Z"
 content="""
Is this a Solaris system?
"""]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/2grhJvAC049fJnvALDXek.6MRZMTlg--#eec89"
 nickname="John"
 subject="comment 2"
 date="2013-01-20T03:38:43Z"
 content="""
No, it's CentOS 6.3 x86_64.,
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.102"
 subject="moreinfo"
 date="2014-03-19T20:49:47Z"
 content="""
What I don't understand about this is, how does `open` fail due to a file being locked? This is Linux, it doesn't have mandatory locking that I know of, and git-annex certianly doesn't use such a thing.

I really need a way to reproduce this and/or a strace. As it is, I've never seen this reported by anyone else and don't understand the failure mode at all.

The relevant part of the code seems to be here:

[[!format haskell \"\"\"
setJournalFile :: JournalLocked -> FilePath -> String -> Annex ()
setJournalFile _jl file content = do
        tmp <- fromRepo gitAnnexTmpMiscDir
        createAnnexDirectory =<< fromRepo gitAnnexJournalDir
        createAnnexDirectory tmp
        -- journal file is written atomically
        jfile <- fromRepo $ journalFile file
        let tmpfile = tmp </> takeFileName jfile
        liftIO $ do
                writeBinaryFile tmpfile content
                moveFile tmpfile jfile
\"\"\"]]

While there is some ctnl locking going on, it locks a special sentinal file, not the file it's writing to.
"""]]


code:
### Please describe the problem.

The command `git annex drop --from` always trusts the local repository, even if
it is marked as untrusted.

### What steps will reproduce the problem?
[[!format sh """
mkdir t u; cd t; git init; git commit --allow-empty -m "Initial commit"; git annex init "Trusted"; date > file; git annex add file; git commit -m "Add file"; cd ../u; git init; git remote add t ../t; git fetch t; git merge t/master; git annex init "Untrusted"; git annex untrust .; git annex get file; cd ../t; git remote add u ../u; git fetch u; cd ..
"""]]

Create two repositories, *t* (trusted) and *u* (untrusted). A file is in both
repositories. When performing `git annex drop file` in repository *t*, `git
annex` will abort because there are not enough copies. But when performing `git
annex drop --from t file` in *u*, git annex will delete the copy.

### What version of git-annex are you using? On what operating system?

Bug was introduced with 6c31e3a8 and still exists in current master (d955cfe7).

### Please provide any additional information below.

The following change seems to solve the problem. (First time working with
Haskell, please excuse the crude code.)

[[!format diff """
diff --git a/Command/Drop.hs b/Command/Drop.hs
index 269c4c2..09ea99a 100644
--- a/Command/Drop.hs
+++ b/Command/Drop.hs
@@ -82,8 +82,9 @@ performRemote key afile numcopies remote = lockContent key $ do
        (remotes, trusteduuids) <- Remote.keyPossibilitiesTrusted key
        present <- inAnnex key
        u <- getUUID
+       level <- lookupTrust u
        let have = filter (/= uuid) $
-               if present then u:trusteduuids else trusteduuids
+               if present && level <= SemiTrusted then u:trusteduuids else trusteduuids
        untrusteduuids <- trustGet UnTrusted
        let tocheck = filter (/= remote) $
                Remote.remotesWithoutUUID remotes (have++untrusteduuids)
"""]]

> [[fixed|done]] --[[Joey]]


code:
### Please describe the problem.
When you rename, move or delete a folder (with files in it) the old folder doesn't get deleted in the other clients

[[!tag moreinfo]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="comment 1"
 date="2013-12-12T19:16:34Z"
 content="""
I can't reproduce this. Tested in direct mode (which I assume you're using, but you didn't say), using both command-line `git annex sync`, and using the assistant. In all cases, when the last file in a directory was removed, the directory was deleted.

Please write back with a version number, and an example showing the problem happening.
"""]]


code:
What steps will reproduce the problem?

Add two remote repositories to same server using ssh. Each directory is added to repositories list and starts syncing.
After both repositories are scanned click on "syncing enabled" to disable syncing for one of the two remote repositories.

What is the expected output? What do you see instead?

Only remote repository which i clicked should stop syncing, but both remote repos start to stop syncing.
Its also working the other way around (enable -> pause -> enable -> ..)

What version of git-annex are you using? On what operating system?

Version: 4.20130314, Debian

Please provide any additional information below.

I am an "webinterface only" user.

[[!tag /design/assistant moreinfo]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-03-16T21:56:37Z"
 content="""
Hmm, I can't quite reproduce this, but I did just fix a bug that, among other manifestations, caused the display of whether syncing is enabled for a remote to be wrong sometimes.

Can you send a copy of your repository's .git/config file to this bug report? (That's `annex/.git/config`)
"""]]


code:
### Please describe the problem.
Yesterday I installed git-annex on two computers and paired their repos.  Today I logged back in to one of them, and as soon as the webapp loaded (autostarted in the background), it popped up an OpenSSH prompt wanting my key's password.  I typed it in, and it popped up another.  This went on several times.  When I hit Cancel instead, it popped up a prompt wanting the password for the user account on my other computer.  Even with that, once wasn't enough.

This is bad enough, but worse is that the password prompt captures the keyboard input so I can't even open my Yakuake console to kill git-annex.  Well, it's difficult and requires hitting Escape rapidly over and over until I can squeeze in a keystroke to the rest of the system.

I don't understand why this is happening.

1.  Shouldn't git-annex have installed a passwordless key on my paired system?  It did that for my remote repo.

2.  The prompt it's using has no option to remember the pasword.  I use ssh-agent, but usually by running ssh-add in a terminal.  Maybe if it would use a prompt that works with the agent it wouldn't ask for the password multiple times.

3.  I think it's opening multiple SSH connections at once, before I've entered the password even once, so even after I enter the password, it will keep asking for it until I've entered it for every SSH process that was already started.

### What version of git-annex are you using? On what operating system?
1 Nov 2013 Linux tarball on Ubuntu Raring 13.04

> [[fixed|done]]; assistant now sets `StrictHostKeyChecking yes`
> when creating ssh remotes. It also fixes up any ssh remotes it already
> created to have that setting (unless StrictHostKeyChecking is already
> being set).
>
> So, when the host key changes, syncing with the remote will now fail,
> rather than letting ssh prompt for the y/n response. In the local
> pairing case, this is completely right, when on a different lan
> and it tries to communicate with the wrong host there. OTOH, if the ssh
> key of a ssh server has really changed, the assistant does not currently
> help dealing with that.
>
> Any ssh remotes not set up by the assistant are left as-is, so this
> could still happen if the ssh host key of such a ssh remote changes.
> I'll assume that if someone can set up their ssh remotes at the command
> line, they can also read the dialog box ssh pops up, ignore the
> misleading "passphrase request" in the title, and see that it's actually
> prompting about a host key change. --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 1"
 date="2013-11-03T00:29:14Z"
 content="""
Pairing sets up the same kind of dedicated passwordless ssh key that is used when adding a ssh server.

It's not clear from your description what program is asking for the password, or even if it's asking for the password for a ssh key. You need to provide more details.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmNu4V5fvpLlBhaCUfXXOB0MI5NXwh8SkU"
 nickname="Adam"
 subject="comment 2"
 date="2013-11-03T00:39:38Z"
 content="""
I'm not sure which program is doing the asking either.  The title bar says \"OpenSSH\".  It is asking for the SSH key password, but if I dismiss the dialog without entering the password, it will open another dialog asking for the user account's password, e.g. me@laptop, indicating it's SSH falling back to password auth.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlHNJ7FpiXJNwSmojlwKwXhhF5QvwpNPmI"
 nickname="Colin"
 subject="Confirmed"
 date="2013-11-14T17:57:23Z"
 content="""
I am seeing the same problem. It happened on a machine running git-annex on Debian Testing some weeks ago, and I eventually disabled the offending repository for my sanity (incidentally - it seems to have vanished so I don't know how to re-enable it apart from adding it again).

It has just very recently (some days) started on this machine running Debian Unstable.

"""]]


code:
[[!comment format=mdwn
 username="https://id.koumbit.net/anarcat"
 ip="2001:1928:1:9::1"
 subject="similar"
 date="2013-12-02T01:39:36Z"
 content="""
I am having a similar issue here. The problem is that the assistant is started along with my regular session, but the SSH keys required for the assistant to sync are prompted during my session startup. So I actually get lots of prompts, because the assistant is started before I have time to enter the password for my SSH key.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkdiXn5s1wX2Zncy1S7_h-yPTv2gt0bmmY"
 nickname="Jacob"
 subject="Same problem on ubuntu"
 date="2013-12-15T14:22:55Z"
 content="""
Same problem, I even manually setup ssh keys between the two servers but it just loops and keeps asking for a password every sync.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="comment 6"
 date="2013-12-19T16:36:55Z"
 content="""
Taking a screenshot, and running `ps -fax` to find what programs are
running, seem like the logical first steps to get anywhere on this
problem.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="comment 7"
 date="2013-12-19T17:54:33Z"
 content="""
Seems likely that this is ssh prompting that the host key has changed. The dialog box has \"password\" at the top, but if you actually read it, it's prompting for a yes/no response to a host key change.

So, I probably need to turn on StrictHostKeyChecking.
"""]]


code:
What steps will reproduce the problem?

1. Click  "Remote server: Set up a repository on a remote server using ssh."
2. Enter hostname and different username than currently logged in user
3. Click check this server

What is the expected output?

> I expected to see the next step in the remote repo creration process.

What do you see instead?

> Failed to ssh to the server. Transcript: ssh_askpass: exec(/usr/libexec/ssh-askpass): No such file or directory Permission denied, please try again. ssh_askpass: exec(/usr/libexec/ssh-askpass): No such file or directory Permission denied, please try again. ssh_askpass: exec(/usr/libexec/ssh-askpass): No such file or directory Permission denied (publickey,password).

What version of git-annex are you using?

> git-annex: Version: 3.20130114

On what operating system?

> OSX: 10.8.2

Please provide any additional information below.

> I mentioned "with a different username" because the assistant will allow me to create a remote repository on the same target machine when I use my normal username. I think this is most likely because I have a ssh-key setup for the account on the remote machine. However I do not want to assume anything and send you down the wrong OSX rabbit hole.

> After a little research it seems that OSX does not have a ssh-askpass

[[!tag /design/assistant/OSX]]
[[!meta title="ssh-askpass not available on OSX"]]

> [[fixed|done]]; the webapp now handles ssh password prompting on its own,
> ssh-askpass is not needed. --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.3.125"
 subject="comment 1"
 date="2013-02-05T19:45:24Z"
 content="""
It should be possible to use SSHPassKey as described here: <http://benno.id.au/docs/ssh_macosx.pml>

However, this all seems pretty crummy, and so perhaps it would be worth it for the webapp to do its own ssh password prompting when setting up a remote.
"""]]


code:
[[!comment format=mdwn
 username="Peng"
 ip="171.37.79.254"
 subject="Found an easy solution"
 date="2013-06-15T03:34:52Z"
 content="""
Just install this script on OS X, and you'll get the ssh-askpass command.

https://github.com/markcarver/mac-ssh-askpass

Tested working successfully on OS X 10.8.3.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnb4NsPsSSHTniueRuVnBzTLrDiDqBZIfY"
 nickname="Doug"
 subject="Same problem but still does not work with askpass installed"
 date="2013-07-24T18:19:25Z"
 content="""
So, I got askpass installed and I'm still getting the same error as everybody else. It's a strange thing as I have my public key dumped on the remote server and I can log in fine manually. It's an empty passphrase so there shouldn't be any prompting going on.
"""]]


code:
### Please describe the problem.

git-annex build on Windows does not complete when user name has a space in it

### What steps will reproduce the problem?

Build git-annex from scratch as described on the Windows page of the guide when running as user Joe Blogs

### What version of git-annex are you using? On what operating system?

Windows 7 Prof, Cygwin install on 28/5/2014

### Please provide any additional information below.

The problem arises in escapeDosPaths of EvilLinker: windows usernames can have spaces in them which also need escaping

[[!tag confirmed]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.176"
 subject="comment 1"
 date="2014-05-28T22:29:05Z"
 content="""
Unfortunately, this is fairly intractable. It would be more productive to fix the [ghc bug](https://ghc.haskell.org/trac/ghc/ticket/8596) that the EvilLinker works around.

Recommend building in a different directory in the meantime.
"""]]


code:
The Date resolution of the FAT filesystem is only 2 seconds for the "last modified time."
This leads to the strange behaviour, that after umount and remount of an usb drive (direct mode) git-annex thinks that suddenly approx. 50% of
the files are modified.  (after remount the "seconds" appears to be rounded to even values - the inode cache before unmount had 1 second resolution) So git-annex is not real "guilty" but it would be fine to create a "workaround" for this problem...

Possible the best solution for this is to set even values for the seconds in the filesystem and in annex internal tables direct after the `git annex get`.
Other solution would be to treat differences up to 1s in modification time as unmodified or create an new parameter like rsync's "modify-window" for this. To do an `git annex sync` or `git annex add` is in my opinion not a good option, because one could add so Bad file content by accident...

Here's an konsole session to show this behaviour:

    $ mount /mnt/transfer/
    $ git clone source/ /mnt/transfer/transfer-repo

    Klone nach '/mnt/transfer/transfer-repo'...
    Fertig.

    $ cd /mnt/transfer/transfer-repo/
    $ git annex init "test"

    init test
      Detected a filesystem without fifo support.

      Disabling ssh connection caching.

      Detected a crippled filesystem.

      Enabling direct mode.
    ok
    (Recording state in git...)

    $ git annex group here transfer

    group here (merging origin/git-annex into git-annex...)
    (Recording state in git...)
    ok
    (Recording state in git...)

    $ git annex wanted here standard

    wanted here ok
    (Recording state in git...)

    $ git annex get --auto

    get n01.mp3 (from origin...)
    SHA256E-s1159018--5674452792970dc03e9ba47d3a8af5ad7c8da6b3ca19e8e64b9a4cf462d4a92d.mp3
         1159018 100%   82.62MB/s    0:00:00 (xfer#1, to-check=0/1)

    sent 1159308 bytes  received 31 bytes  2318678.00 bytes/sec
    total size is 1159018  speedup is 1.00
    ok

    get n02.mp3 (from origin...)
    SHA256E-s1622113--03998dc10c4839d5ab9aeaceaa63f0363c9d728aaaca2a2707f025c7b9e920a3.mp3
     1622113 100%   34.45MB/s    0:00:00 (xfer#1, to-check=0/1)

    sent 1622459 bytes  received 31 bytes  3244980.00 bytes/sec
    total size is 1622113  speedup is 1.00
    ok

    ...
    ...

    –--> All 29 files (n01.mp3 to n29.mp3) successfully got

    (Recording state in git...)

    $ git annex status
    $ stat * >../stat-before-umount
    $ cd /
    $ umount /mnt/transfer
    $ mount /mnt/transfer
    $ cd /mnt/transfer/transfer-repo
    $ stat * >../stat-after-remount
    $ git annex status
    M n05.mp3
    M n10.mp3
    M n11.mp3
    M n13.mp3
    M n16.mp3
    M n17.mp3
    M n20.mp3
    M n22.mp3
    M n23.mp3
    M n24.mp3
    M n26.mp3
    M n27.mp3
    $ diff -u ../stat-before-umount ../stat-after-remount | grep -B8 "+Modifiziert" | grep -E "Datei:|Modifi"

       Datei: „n05.mp3“
    -Modifiziert: 2014-05-03 19:42:39.000000000 +0200
    +Modifiziert: 2014-05-03 19:42:38.000000000 +0200

       Datei: „n10.mp3“
    -Modifiziert: 2014-05-03 19:43:05.000000000 +0200
    +Modifiziert: 2014-05-03 19:43:04.000000000 +0200

       Datei: „n11.mp3“
    -Modifiziert: 2014-05-03 19:43:07.000000000 +0200
    +Modifiziert: 2014-05-03 19:43:06.000000000 +0200

      Datei: „n13.mp3“
    -Modifiziert: 2014-05-03 19:43:15.000000000 +0200
    +Modifiziert: 2014-05-03 19:43:14.000000000 +0200

       Datei: „n16.mp3“
    -Modifiziert: 2014-05-03 19:43:21.000000000 +0200
    +Modifiziert: 2014-05-03 19:43:20.000000000 +0200

       Datei: „n17.mp3“
    -Modifiziert: 2014-05-03 19:43:29.000000000 +0200
    +Modifiziert: 2014-05-03 19:43:28.000000000 +0200

       Datei: „n20.mp3“
    -Modifiziert: 2014-05-03 19:43:53.000000000 +0200
    +Modifiziert: 2014-05-03 19:43:52.000000000 +0200

       Datei: „n22.mp3“
    -Modifiziert: 2014-05-03 19:44:13.000000000 +0200
    +Modifiziert: 2014-05-03 19:44:12.000000000 +0200

       Datei: „n23.mp3“
    -Modifiziert: 2014-05-03 19:44:23.000000000 +0200
    +Modifiziert: 2014-05-03 19:44:22.000000000 +0200

       Datei: „n24.mp3“
    -Modifiziert: 2014-05-03 19:44:31.000000000 +0200
    +Modifiziert: 2014-05-03 19:44:30.000000000 +0200

       Datei: „n26.mp3“
    -Modifiziert: 2014-05-03 19:44:35.000000000 +0200
    +Modifiziert: 2014-05-03 19:44:34.000000000 +0200

       Datei: „n27.mp3“
    -Modifiziert: 2014-05-03 19:44:39.000000000 +0200
    +Modifiziert: 2014-05-03 19:44:38.000000000 +0200

> fixed [[done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="martin"
 ip="193.174.111.250"
 subject="specification"
 date="2014-06-11T05:43:02Z"
 content="""
What you suggested (to not add if exact 1s or exact 1h time difference and changed checksum) would do the trick but it's hard for the user to test if this really works as expected and it would be more secure and IMHO more clear to do it like this:

If `git annex add` \"meets\" such files, `git annex add` should not really add / commit these pseudo-modified files but *only adapt the timestamps* (which make the software erroneously thinks the files were modified) to the vfat values. (if checksum is still correct) I guess the timestamps are in the annex branch.

`git annex add` could - instead of write: \"add ok\" - only write: \"corrected timestamp ok - no need to add\"

So the user sees whats really happen and after this these files appears again as what they are: unmodified. `git annex status` would say nothing (everything fine), and `git annex fsck` would checksum these files again.

I suppose that the way i suggest is harder to code :-( Unfortunately I'm not a good coder and not experienced in haskell for sending a patch myself / the open source way...

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 11"
 date="2014-06-11T17:22:12Z"
 content="""
@martin, to the best of my knowledge and testing, what you're proposing `git annex add` do in this case is identical to what it already happens to do, except for the message displayed.

Do you have an example of it not doing that?

We seem to be talking past one-another, since I've now said three times this is what it does. And I've tested it twice.

I don't think that the current behavior of add, or your suggested behavior (which again, happens to be nearly identical) is sufficient to close this bug report with.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 12"
 date="2014-06-11T18:05:48Z"
 content="""
The resolution problem does not seems to affect Windows, at least not on NTFS. In my tests, the mtime is fully preserved across reboots there.

However, any change to the timezone on Windows does manage to mess up all the timestamps. Presumably this same flawed approach is used for DST adjustments.

Example from inode cache after a 1 hour time zone change, which forced git-annex add to re-checksum:

<pre>
--1 2221 1395843799
+-1 2221 1395847399
</pre>

Of course, not all time zones are 1 hour offset, so as a heuristic, treating timestamps +- 60*60*Int as the same, would be pretty bad. Instead, it would probably make sense, on windows, to normalize the timestamp, using the current time zone, to get to the UTC timestamp. (Of course on unix, a file's timestamp is always given in UTC.)

Unfortunately, Data.Time.LocalTime.getCurrentTimeZone doesn't seem to really work on windows. It always returns UTC+1 in my tests.

Anyway, I'm now looking at this as two separate problems, the Windows Time Zone Sucks problem and the FAT Metadata Sucks problem. They will probably have different solutions..
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="FAT MetaData Sucks: an approach"
 date="2014-06-11T18:15:59Z"
 content="""
git-annex already deals with FAT metadata sucking by using the inode sentinal file, to detect when a FAT filesystem has been remounted with new random inodes, and so ignore apparent inode changes.

So, when a InodeCache is compared in weak mode due to that, it could just treat all mtimes within 2 seconds as the same. This would limit the brain-damange to FAT.

One problem with this idea is that it's specific to linux's handling of FAT, with its random inodes. A FAT mounted on windows will have -1 for the inodes across remounts. So this method can't detect if a filesystem on windows is FAT, and has the problem, or NTFS and not.

But, I think the FAT side of this problem is also linux-specific. Linux dummies up good metadata for FAT, and then has to throw it away/degrade on unmount. Windows presumably uses real timestamps, with low resolution on FAT from the beginning.

I don't know how eg OSX handles this. If it used constant inodes but cached higher resolution mtimes for FAT, this approach would not work there.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 14"
 date="2014-06-11T18:59:49Z"
 content="""
Opened a separate bug for [[Windows_file_timestamp_timezone_madness]].

Fixed FAT issue on Linux as discussed in my previous comment.
"""]]


code:
[[!comment format=mdwn
 username="martin"
 ip="89.183.21.108"
 subject="In reply to comment 11"
 date="2014-06-12T09:22:53Z"
 content="""
Dear joey,

thanks a lot for fixing this!

Only for this protocol (not to have the last word ;-)

What `git annex add` already does now and what i suggested it could do better is indead the same (result) for uncorrupted files (the frequent case).
But i'am afraid, it could be slightly different if we deal with corrupted files. The preset behaviour would add this file, the suggested behaviour would not...

But this discussion is history now - i'm lucky :-))

Now i do some tests with daylight savings time change - hopefully there are no problems!
"""]]


code:
[[!comment format=mdwn
 username="zardoz"
 ip="92.227.51.179"
 subject="comment 1"
 date="2014-05-13T16:50:57Z"
 content="""
A related issue is that a change from/to daylight-saving time will make files appear to be one hour older/younger. A modify-window couldn’t acommodate that. VFAT is really a huge pain…
"""]]


code:
[[!comment format=mdwn
 username="martin"
 ip="89.183.46.169"
 subject="Possible solution?"
 date="2014-05-13T17:51:41Z"
 content="""
How about this quick'n dirty vfat compromise:

For vfat only we do like this (at least for `git annex fsck` command, so that the user doesn't wonder about the strange effects of vfat and can repair this):

if we have an exact time difference of 1s (probably \"inode problem\") or 1h (\"utc problem\")
we treat this file as likely unmodified and check this via the normal checksum algorithm.

if checksum is ok,  we give a message to the user, that the \"file has only a vfat timestamp problem\" but has correct checksum and if the user decides to do so git annex sets the timestamp in the filesystem to the value from annex' internal tables...

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 3"
 date="2014-06-05T17:06:40Z"
 content="""
> if we have an exact time difference of 1s (probably \"inode problem\") or 1h (\"utc problem\") we treat this file as likely unmodified and check this via the normal checksum algorithm.

That sort of makes sense, but when is git-annex supposed to do that?

If `git annex add`, it already checksums the file, and already stages no change if the file's checksum is the same. And if the user has told git-annex to add the file and it's changed, the presumption is they know it's changed and want to add the new version.

> To do an git annex sync or git annex add is in my opinion not a good option, because one could add so Bad file content by accident...

If not in add or sync, then when?

----

I am actually having a hard time coming up with a scenario where this problem results in any more than extra checksumming work by git-annex.

The only scenario I see is: The drive is unmounted, gets corrupted, is remounted, and this timestamp nonsense causes git-annex to think a file (that has already gotten corrupted) has in fact changed, so it commits the corrupted version.
"""]]


code:
[[!comment format=mdwn
 username="martin"
 ip="193.174.111.250"
 subject="It confuses users"
 date="2014-06-05T19:05:12Z"
 content="""
Other scenario: User thinks, that his usb drive has a problem, because approx. 50% of the files are gone/changed without obvious reason. I spent several hours to find the reason for this... I think we should at least give kind of explanation/warning message to the user
"""]]


code:
[[!comment format=mdwn
 username="martin"
 ip="193.174.111.250"
 subject="another scenario"
 date="2014-06-05T19:52:04Z"
 content="""
If a user uses such a crippled filesystem as a transfer repo and he does an `git-annex get --auto` on the target pc for getting the content he gets only 50% of the content. (user gets only a strange message that git-annex cannot access the repo or something like this) The user does not have the intuition to do an extra `git-annex add` to get the content because from his perspective he did not made any changes... (sorry for my bad english - i hope you know what i mean)
"""]]


code:
[[!comment format=mdwn
 username="martin"
 ip="89.183.56.177"
 subject="use case and answer to joey's question "...then when...""
 date="2014-06-08T04:59:42Z"
 content="""
Hi joey,

i really think we should repair the timestamp instead of force the user to `git-annex add` possibly corrupted files/content (see your comment above) git-annex checksums are excellent for data integrity but they are useless if we bypass them in case of adding and propagating
potentially corrupted content with `git-annex add`

So here's for example a useful use case:

1. user fills his transfer repo in town A. He checks (`git annex fsck`)  that everything is fine. He unmounts the drive and travels to town B.

2. In town B user mounts the drive and sees, that he suddenly doesn't have \"access\" to those \"crippled files\" on his transfer repo (the files seems modified for the user without a reason - why should he `git-annex add` them again?? - he'd rather think about data corruption) . He wonders whats going on and thats why he does a

3. `git-annex fsck` . `git annex fsck` should checksum also such crippled files, should report correct checksums if they are correct (so that the users knows their usb drive is working properly) and should give a message like this: \"checksum ok - crippled timestamp - repair with git-annex fsck --repair-timestamp or define a modify window \"- (to be implemented - in rsync the problem is already solved with this option \"--modify-window=NUM     compare mod-times with reduced accuracy\")

In my case distance between town B and town A was approx 400km and in town B i didnt know what was going on. So i went back to town A without success for further investigation of my usb drive... :-((

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 7"
 date="2014-06-09T19:29:09Z"
 content="""
A transfer repository is normally bare, so would not have this problem.
"""]]


code:
[[!comment format=mdwn
 username="martin"
 ip="89.183.78.73"
 subject="`git-annex add` (called by sync) should do the job and bring the files back home (IMHO)"
 date="2014-06-10T15:58:25Z"
 content="""
If one does an `git annex sync` these crippled-pseudo-modified-files are *automatically* `git annex add`ed

   git annex status
   M datei1
   M datei5
   M datei6
   git annex sync
   commit  add datei1 ok
   add datei5 ok
   add datei6 ok

To avoid the risk of adding and propagating potentially corrupted content `git-annex add` should
\"simply\" correct the timestamps (adjust to the new even inode values) for files with correct checksum but timestamp
difference of 1s or 1h

With this procedure i would have better sleep with this personal second use case:

repo1: on the computer (direct mode - client)
repo2: on usb stick -  (direct mode - client - vfat - music for car)

From time to time mp3 files are transferred to the stick. Then stick
goes to the car and after some days back to the computer to be
synchronized again. Everytime approx. 50% of the recently new added files are
added again (via sync) because of these nonsens timestamps.

So i think, the clean solution is to correct only the timestamps instead
of adding again possibly corrupted files. If we dont do this users adapt to
add files again and again (they need not to be added) and one day they
add corrupted content. Like on windows (tm) you klick OK and OK again and
here you add again and again and one day one add once too much ;-)

Excuse me for this long comment...

P.S. git annex is an ingenious piece of software - thanks a lot for this joey!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 9"
 date="2014-06-10T19:27:55Z"
 content="""
> To avoid the risk of adding and propagating potentially corrupted content git-annex add should \"simply\" correct the timestamps (adjust to the new even inode values) for files with correct checksum but timestamp difference of 1s or 1h

Since git-annex add already does that in this case, I think what you're actually suggesting is that, if the timestamp has a 1s or 1h diff, but the checksum has changed, git-annex add should refuse to add the files, on the grounds that it appears to be corrupt. Which obviously fails badly in at least the 1h case, because someone could add a file, then wait 1 hour and git-annex add a modified version.
"""]]


code:
### Please describe the problem.
I've a repository with a prefered content of "exclude=cours/vieux/* and ( include=cours/* or include=Programmes/* or include=Stage/* )", but there is a repository that has a (identical) copy both in cours/vieux and in Stage/ so git annex sync --content will
- get the content in Stage/
- then drop it because it is in cours/vieux/

And again, and again...

### What version of git-annex are you using? On what operating system?
git-annex 5.20140127 in debian/sid

> This is actually a duplicate of
> [[Handling_of_files_inside_and_outside_archive_directory_at_the_same_time]].
> As noted in that bug report, this should already be fixed in direct mode,
> but still happens in indirect mode when the full scan is done (at
> startup, and occasionally at other times).
>
> [[dup|done]] --[[Joey]]


code:
**What steps will reproduce the problem?**

Create two annexes from the command line on two separate machines:

    mkdir ~/Files.annex
    cd ~/Files.annex
    git init
    git annex init
    git annex untrust .
    git annex direct

Add remote to each one pointing to the other:

    git remote add [remote] [remote hostname]:Files.annex

Start assistant on both repos:

    git annex assistant

Fill one repository with a few text files, and wait for them to propagate.

Edit one of the text files using vim, and save.

**What is the expected output? What do you see instead?**

Edited file should remain in the repo, but a significant portion of the time, the file disappeared from the repo in which it was edited (the file is present and properly synced on the other repo).

**What version of git-annex are you using? On what operating system?**

git-annex 4.20130323, Mac OS X on the repo where the file was edited, Arch Linux for the other paired repo.

**Please provide any additional information below.**

I have observed this problem setting up the repos through the webapp as well, so I don't think it is related to setting up the repos manually. I think the way vim is writing the files seems to be tickling a race condition (both command-line vim and MacVim produce the behavior). I started trying to work around it by switching to emacs to edit those files, and the files haven't disappeared from the edited repo (so far at least).

[[!tag /design/assistant moreinfo]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-04-08T17:14:23Z"
 content="""
When you say that the file \"disappeared from the repo\", do you mean that the actual file is no longer present on disk, or that the file is present on disk, but is no longer staged in the git repository?

Version 4.20130405 fixes a bug that can result in the latter behavior.
"""]]


code:
### Please describe the problem.
Using the webapp to generate a new (local) repository instantly takes it to the following state:
[[!format sh """
user@local:~/Annex$ git status
# On branch master
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#	deleted:    uuid.log
#
user@local:~/Annex$ git branch
  git-annex
* master
user@local:~/Annex$ git log
commit 90bfcaf17b0576d8ecdc48ae44dda22d41464acc
Author: local  <user>
Date:   Sun Nov 3 15:30:17 2013 +0100

    created repository
user@local:~/Annex$ git show HEAD
commit 90bfcaf17b0576d8ecdc48ae44dda22d41464acc
Author: local  <user>
Date:   Sun Nov 3 15:30:17 2013 +0100

    created repository

diff --git a/uuid.log b/uuid.log
new file mode 100644
index 0000000..9df3670
--- /dev/null
+++ b/uuid.log
@@ -0,0 +1 @@
+987e7b9a-aa9d-41db-ae92-23fcae7f6187 user@local:~/Annex timestamp=1383489017.181
user@local:~/Annex$
"""]]

I'm new to git-annex, so I'm not quite sure, but looking at [[internals]] this file should only exist in the git-annex branch, not in master. Furthermore, from this state it seems impossible to get "sync with your other devices" to work, because of a merge conflict on this change.

Perhaps some sort of a race-condition with the annex-assistant picking up the uuid.log file while the repository is being initialized?

### What version of git-annex are you using? On what operating system?
Ubuntu 13.10 with git-annex 4.20130815

> [[fixed|done]]; see comments. (This fix needs to be backported to Ubuntu.) --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 1"
 date="2013-11-03T16:48:25Z"
 content="""
I can't reproduce this at all. What version of git do you have installed? Did you install git-annex from ubuntu's repository? Does the same thing happen if you install the standalone linux tarball and use it to make a new repository?

git-annex never creates a file named uuid.log on disk, so it's quite strange that it shows up in the initial commit to the master branch. It sort of looks like somehow git-annex's normal use of a separate index file to stage the uuid.log to the git-annex branch is failing. Since I have never seen any problem with that, I have to suspect that the ubuntu build is somehow badly broken. Or that the git in Ubuntu is for some reason ignoring `GIT_INDEX_FILE`.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 2"
 date="2013-11-03T17:02:47Z"
 content="""
I made an Ubuntu saucy chroot, apt-get installed git-annex from universe, and ran the webapp in there. I did not reproduce this problem.

The cause of the problem, it seems, must be something local to your system. Perhaps you have an environment variable set that is messing up git. Or perhaps you have a different, broken version of git installed.

Can you \"git show git-annex\" in the repository? It should show a commit made to the git-annex branch that adds the uuid.log there.
"""]]


code:
[[!comment format=mdwn
 username="tanen"
 ip="83.128.159.25"
 subject="comment 3"
 date="2013-11-03T17:35:00Z"
 content="""
Very strange: this is on a machine that I wiped and reinstalled just a few hours ago, it's a completely fresh Ubuntu 13.10 install with barely anything installed but the defaults. Git version is 1.8.3.2-1

I initially just pulled git-annex from the Ubuntu repo. After that I grabbed a more recent version from https://launchpad.net/ubuntu/+source/git-annex/4.20131101/+build/5189754 which is showing the same behavior.

\"git show git-annex\" indeed shows the commit creating the uuid.log file on the git-annex branch. master has just one commit, with description \"created repository\" and creates a \"uuid.log\" file. The contents of the master uuid.log are identical to the one in the git-annex branch.

I'm currently in the middle of trying out a git-annex setup so I can't switch versions again right now, but given the above I imagine a fresh 13.10 VM should show the same behavior.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 4"
 date="2013-11-06T15:09:19Z"
 content="""
Intriguing -- I was able to reproduce this bug after installing the Ubuntu server ISO in a VM.

Which is really strange, the only difference between this and my debootstrapped chroot should be the kernel..
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 5"
 date="2013-11-06T16:27:49Z"
 content="""
Running the prebuilt tarball build of git-annex, the bug does not occur.

However, if I remove the git shipped with the prebuilt tarball, so it uses the system git, I do see the bug. So, it's apparently git version dependent.

Also, I was able to reproduce it in a amd64 chroot. My other chroot was i386. Somehow architecture specific bug?

---

Instrumenting all calls to git to be logged with the full environment and command, I found this:

GIT_INDEX_FILE='/home/foo/annex/.git/annex/index'
--git-dir=/home/foo/annex/.git --work-tree=/home/foo/annex commit --quiet --allow-empty -m created repository

This certainly looks like the index file setting for the git-annex branch is somehow leaking out past the branch commit operations. It continued setting that while setting up `gc.auto`; the next call to git after that stopped setting the index file.

The only way I can see offhand this could possibly happen is due to an exception. It may be that on ubuntu an exception is thrown by code that runs a git command with the index file set, for whatever reason, and this causes the code that normally resets it back to not run.

----

Ok, found it!

\"withIndex entered\"

*** Please tell me who you are.

Run

  git config --global user.email \"you@example.com\"
  git config --global user.name \"Your Name\"

to set your account's default identity.
Omit --global to set the identity only in this repository.

fatal: unable to auto-detect email address (got 'foo@darkstar.(none)')
\"withIndex entered\"
\"withIndex cleaned up\"

Note lack of clean up after the first withIndex call. Thus leaving the environment passed to git polluted for further calls.

This also explains why it's only happening on some systems, or with some versions of git. git's got all kinds of complexity around its username and email handling code.

I have fixed this in git.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.47"
 subject="comment 6"
 date="2013-11-06T16:40:57Z"
 content="""
Ubuntu bug report about this: <https://bugs.launchpad.net/ubuntu/+source/git-annex/+bug/1248623>

It should be pretty easy to backport the fix to the version in Ubuntu. The relevant git commits are ee23be55fd3e7e202bc721c124f78b79d1aba6df and 81117e8a9d19d4739d3773d0515006e1ea41c266
"""]]


code:
`GIT_DIR` support isn't right. Git does not look for `GIT_DIR/.git`;
git-annex does.

Also, to support this scenario, support for core.worktree needs to be added
as well:

	mkdir repo workdir
	git --work-tree=$PWD/workdir --git-dir=$PWD/repo init
	export GIT_DIR=$PWD/repo
	git status
	# ok
	git annex init "new repo"
	# fail

--[[Joey]]

> [[fixed|done]] --[[Joey]]


code:
### Please describe the problem.
When I try to create a megaannex remote with pubkey encryption GPG complains about not finding the public key.

### What steps will reproduce the problem?
See below

### What version of git-annex are you using? On what operating system?
5.20140129 under OSX.

### Please provide any additional information below.

[[!format sh """
% USERNAME="<username>" PASSWORD='<password>' git annex -vd initremote mega type=external externaltype=mega encryption=pubkey keyid=X folder=Documents mac=HMACSHA512
[2014-02-06 11:39:14 CET] read: git ["--git-dir=/Users/dxtr/Documents/.git","--work-tree=/Users/dxtr/Documents","show-ref","git-annex"]
[2014-02-06 11:39:14 CET] read: git ["--git-dir=/Users/dxtr/Documents/.git","--work-tree=/Users/dxtr/Documents","show-ref","--hash","refs/heads/git-annex"]
[2014-02-06 11:39:14 CET] read: git ["--git-dir=/Users/dxtr/Documents/.git","--work-tree=/Users/dxtr/Documents","log","refs/heads/git-annex..62dc22cced06268fa5adcf54992eb1169c6ca1aa","--oneline","-n1"]
[2014-02-06 11:39:14 CET] chat: git ["--git-dir=/Users/dxtr/Documents/.git","--work-tree=/Users/dxtr/Documents","cat-file","--batch"]
initremote mega (encryption setup) [2014-02-06 11:39:14 CET] read: gpg ["--batch","--no-tty","--use-agent","--quiet","--trust-model","always","--with-colons","--list-public-keys","46726B9A"]
[2014-02-06 11:39:14 CET] read: gpg ["--batch","--no-tty","--use-agent","--quiet","--trust-model","always","--gen-random","--armor","2","256"]
[2014-02-06 11:39:14 CET] chat: gpg ["--batch","--no-tty","--use-agent","--quiet","--trust-model","always","--recipient","X","--encrypt","--no-encrypt-to","--no-default-recipient","--force-mdc","--no-textmode"]
(pubkey crypto with gpg key X) [2014-02-06 11:39:15 CET] chat: git-annex-remote-mega []
[2014-02-06 11:39:15 CET] git-annex-remote-mega --> VERSION 1
[2014-02-06 11:39:15 CET] git-annex-remote-mega <-- INITREMOTE
[2014-02-06 11:39:15 CET] git-annex-remote-mega --> GETCONFIG encryption
[2014-02-06 11:39:15 CET] git-annex-remote-mega <-- VALUE pubkey
[2014-02-06 11:39:15 CET] git-annex-remote-mega --> GETCONFIG folder
[2014-02-06 11:39:15 CET] git-annex-remote-mega <-- VALUE Documents
[2014-02-06 11:39:15 CET] git-annex-remote-mega --> SETCREDS mycreds <username> <password>
(gpg) [2014-02-06 11:39:15 CET] chat: gpg ["--batch","--no-tty","--use-agent","--quiet","--trust-model","always","--decrypt"]
[2014-02-06 11:39:15 CET] chat: gpg ["--batch","--no-tty","--use-agent","--quiet","--trust-model","always","--batch","--encrypt","--no-encrypt-to","--no-default-recipient","--force-mdc","--no-textmode"]
gpg: no valid addressees
gpg: [stdin]: encryption failed: no such user id

git-annex: user error (gpg ["--batch","--no-tty","--use-agent","--quiet","--trust-model","always","--batch","--encrypt","--no-encrypt-to","--no-default-recipient","--force-mdc","--no-textmode"] exited 2)
failed
git-annex: initremote: 1 failed

# End of transcript or log.
"""]]

> [[fixed|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkzwmw_zyMpZC9_J7ey--woeYPoZkAOgGw"
 nickname="dxtrish"
 subject="comment 1"
 date="2014-02-06T11:00:49Z"
 content="""
I just confirmed it works with encryption=hybrid
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="71.80.94.56"
 subject="comment 3"
 date="2014-02-06T21:53:23Z"
 content="""
I can reproduce this, but only when using the hook special remote, so it's some problem with it.
directory special remote works ok.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="71.80.94.56"
 subject="comment 3"
 date="2014-02-06T22:07:55Z"
 content="""
Actually, it seems to affect also S3 and other remotes that call setCreds.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkzwmw_zyMpZC9_J7ey--woeYPoZkAOgGw"
 nickname="dxtrish"
 subject="comment 4"
 date="2014-02-07T11:11:23Z"
 content="""
Could it be some general issue with setCreds?

The reason I'm wondering is that megaannex isn't really working. It doesn't seem to (properly) get the credentials so it's acting funny and getting an Invalid Argument exception where, according to what I understand from the documentation, it shouldn't be possible to get an Invalid Argument exception.

Ofcourse I have contacted the maintainer of megaannex (TobiasTheViking on github) too. I've also started looking in to it myself.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkzwmw_zyMpZC9_J7ey--woeYPoZkAOgGw"
 nickname="dxtrish"
 subject="comment 5"
 date="2014-02-07T11:12:24Z"
 content="""
Oh, sorry. I didn't see that you had already fixed it.
"""]]


code:
#### What steps will reproduce the problem?

1. Create a new repository with a directory
2. Add files
3. Select "Store your data in the cloud" with the "Remote server" option
4. Enter host, user, directory
5. Select "Use an encrypted rsync repository on the server" (Will there be an option for unencrypted later?)
6. GPG Passphrase prompt comes up for every file

#### What is the expected output? What do you see instead?

I expect to enter a passphase once and then it will sync all files with the remote server.

Instead, it begins syncing the files to the server but prompts for a GPG passphase for every single file.

#### What version of git-annex are you using? On what operating system?

3.20121017 precompiled binary on Arch Linux

#### Please provide any additional information below.

Not sure if I'm just missing a setting for GPG, but I would think I should only need to use the web app to configure the remote server.

[[!tag /design/assistant]]
[[!tag confirmed]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.194"
 subject="you should install a gpg agent"
 date="2012-10-31T17:39:53Z"
 content="""
A gpg agent will cache your passphrase. It is beyond the scope of the git-annex package to provide one (though it does bundle gpg), but it should be easy to install gpg-agent on your distribution.

That's all that's needed for normal git-annex use, but the assistant does seem to have a larger problem in this area, since it can need to unlock a remote's key at any time to sync files from it. Since a separate
process is spawned for each transfer, this defeats git-annex's normal in-process caching of encryption keys of remotes. So I think it needs to unlock any encrypted special remotes at startup, or when first accessing them, and pass the cached keys to the transfer processes it spawns. This is now on my todo list.

However, none of the special remotes set up by the assistant will use
password protected gpg keys, even when it's using encryption it's using a
non-password protected shared key. So only encrypted special remotes set up
at the command line cause this problem.
"""]]


code:
I recently had my git-annex repository die and it needed to be repaired. Two of my repositories are external hard drives. When I tried to use git-annex repair, it would churn for some hours, then error because the external hard drives were not plugged in. When I brought the two hard drives home from the various places that they are (safely) stored, it all worked fine, but it would have been great if git-annex repair could somehow do what it could with what was connected and do the rest as and when the other drives are plugged in. This must only become more of a problem as git-annex is used for longer, as one may have a handful of USB keys storing a little on each.

[[!taglink moreinfo]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.244"
 subject="comment 1"
 date="2014-04-07T19:25:25Z"
 content="""
Repair does not require access to remotes, but it will certianly yield a better result to have one remote available. Otherwise, information that cannot be repaired will be missing from the repository. Repair will still successfully complete in this situation though.

If you have an error message, please paste it.
"""]]


code:
### Please describe the problem.

I was playing around thinking to try out git-annex, and bup

I sucessfully added 6GB of images in subdirectories in ~/Pictures, so decided to push over another dir full -

more than 30GB more...

sven@quad:~/Pictures$ git annex add .
....
add 2012/01/10/20120110-024457/IMG_1830.JPG
git-annex: <file descriptor: 7>: hFlush: resource vanished (Broken pipe)
failed
add 2012/01/10/20120110-024457/IMG_1831.JPG
git-annex: <file descriptor: 7>: hFlush: resource vanis
sven@quad:~/Pictures$
sven@quad:~/Pictures$ git annex add .
Stack space overflow: current size 8388608 bytes.
Use `+RTS -Ksize -RTS' to increase it.
sven@quad:~/Pictures$ git git commit -a -m added
git: 'git' is not a git command. See 'git --help'.

Did you mean this?
        init
sven@quad:~/Pictures$ git commit -a -m added
Bus error (core dumped)

### What steps will reproduce the problem?
see above

### What version of git-annex are you using? On what operating system?
sven@quad:~/Pictures$ git annex version
git-annex version: 3.20120406
local repository version: 3
default repository version: 3
supported repository versions: 3
upgrade supported from repository versions: 0 1 2
sven@quad:~/Pictures$ uname -a
Linux quad 3.8.0-34-generic #49~precise1-Ubuntu SMP Wed Nov 13 18:05:00 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

# End of transcript or log.
"""]]

> Ancient git-annex version. Doubt it affects current version. [[!tag moreinfo]] --[[Joey]]

>> Actually, this is a dup of [[runs_of_of_memory_adding_2_million_files]] so [[done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.43"
 subject="comment 1"
 date="2014-01-13T19:22:12Z"
 content="""
It looks like git-annex was leaking memory, and so ran out and died.

But, you're using an ancient version of git-annex from 2012. Upgrade to a current version and write back if it still has a memory leak. I doubt it does.
"""]]


code:
Hi,
I already have told about that in a comment here <http://git-annex.branchable.com/bugs/submodule_path_problem/#comment-e86330d15b714a41a07b6548fbc79bb2>, but I am not sure it will be seen.

Then here is an official bug report.
### Please describe the problem.

I have a problem with submodules when the git repository is not a submodule everywhere.

For instance, if A is a git annexed repository and B another git repository. If B adds A as submodules. The symlinks added in A as submodule won't work in the original A.

### What steps will reproduce the problem?

    # creating the master repository
    mkdir annex_master
    cd annex_master/
    git init
    # hack: adding a file to create the master branch
    touch start
    git add start
    git commit -m "start"
    cd ..
    # create another repository
    mkdir annex_sub
    cd annex_sub/
    git init
    # hack: adding a file to create the master branch
    touch start
    git add start
    git commit -m "start"
    # it is a annexed repository
    git annex init sub
    # add the other repository as submodule of the master one
    cd ../annex_master/
    git submodule add ../annex_sub/ module
    cd module/
    git annex init sub_module
    git annex sync origin
    # add an annexed file
    echo test > test
    git annex add
    git annex sync
    # go back to the origin repository
    cd ../../annex_sub/
    git annex sync
    ls -l
This returns test -> ../.git/modules/module/annex/objects/w8/pv/SHA256E-s5--f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/SHA256E-s5--f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2

Actually, the file committed is correct. But the fact it points to '../.git/modules/module/...' makes the link work only if the repository is also a submodule and if this submodule is also located in the modules folder in the parent git repository.

I tried playing with making the repository direct and then indirect, hoping that would fix the symlinks, but it did not work.
### What version of git-annex are you using? On what operating system?

    $ git-annex version
    git-annex version: 4.20130802
    ...

    $ git --version
    git version 1.8.3.2

	$ uname -a
	Linux konixwork 3.9-1-amd64 #1 SMP Debian 3.9.8-1 x86_64 GNU/Linux

### Please provide any additional information below.

[[!tag confirmed]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.63"
 subject="comment 1"
 date="2013-08-24T18:49:18Z"
 content="""
git-annex assumes that it can make a stable symlink from a file in the working tree to a file in the .git directory. There are several ways to break this. One, as noted, is sometimes using a repository as a submodule, and sometimes not. Another would be to play around with `GIT_DIR`.

I don't see a way git-annex can support those use cases, at least in indirect mode.

It does seem like, in direct mode, it should just work. git-annex will commit various symlinks to git, but these symlinks will never be followed to get at the content of a file, since direct mode arranges for the content to be directly present in the working tree.
"""]]


code:
[[!comment format=mdwn
 username="konubinix"
 ip="82.243.233.186"
 subject="Thanks"
 date="2013-08-26T06:25:19Z"
 content="""
Thanks for the reply.

Also thanks for this great tool (Though I am not sure I truelly realize the true power of git annex yet).
"""]]


code:
### Please describe the problem.
hasKey check fails when using a Glacier special remote with a non-default region setting.

### What steps will reproduce the problem?
1. Configure a Glacier special remote with a region other than us-east-1
2. Move some files to the remote
3. Somehow let galcier-cli cache expire. In my case this was waiting for more than 60 hours.
4. Try to copy the file to the glacier remote again.

### What version of git-annex are you using? On what operating system?
5.20140227 on Ubuntu 12.04

### Please provide any additional information below.

[[!format sh """
user@server:$ git annex copy my_file.txt --to glacier
copy my_file.txt (checking glacier...) Traceback (most recent call last):
  File "/bin/glacier", line 730, in <module>
    App().main()
  File "/bin/glacier", line 716, in main
    self.args.func()
  File "/bin/glacier", line 620, in archive_checkpresent
    wait=self.args.wait)
  File "/bin/glacier", line 442, in _vault_sync
    vault = self.connection.get_vault(vault_name)
  File "/usr/local/lib/python2.7/dist-packages/boto/glacier/layer2.py", line 82, in get_vault
    response_data = self.layer1.describe_vault(name)
  File "/usr/local/lib/python2.7/dist-packages/boto/glacier/layer1.py", line 195, in describe_vault
    return self.make_request('GET', uri)
  File "/usr/local/lib/python2.7/dist-packages/boto/glacier/layer1.py", line 118, in make_request
    raise UnexpectedHTTPResponseError(ok_responses, response)
boto.glacier.exceptions.UnexpectedHTTPResponseError: Expected 200, got (404, code=ResourceNotFoundException, message=Vault not found for ARN: arn:aws:glacier:us-east-1:111111111111:vaults/myvault)
(user error (glacier ["archive","checkpresent","myvault","--quiet","SHA256E-s1111111--aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.JPG"] exited 1)) failed
git-annex: copy: 1 failed
"""]]

> [[fixed|done]]; made it use the same glacierParams as everything else.
> --[[Joey]]