code:
### Please describe the problem.

Other references:

https://github.com/basak/glacier-cli/pull/19
http://git-annex.branchable.com/special_remotes/glacier/#comment-a2b05b8dc2d640ee498d90398f02931c

#### Background

 * Glacier doesn't support keys that the client selects, unlike S3. If you upload to Glacier, Glacier assigns a unique ID, not the client.
 * Glacier does support an "archive description" which is immutable. It also provides this "archive description" in an inventory listing, together with the unique IDs.
 * An "archive description" is not a unique key. It's perfectly possible to upload multiple archives to Glacier with the same "archive description".
 * glacier-cli uses the "archive description" field as an upload identifier, since the unique IDs are unfriendly to users. However, since they are potentially ambiguous identifiers, it also supports disambiguation using the ID itself. See "Addressing Archives" in README.md for details.

#### The Problem

This what I believe is happening in the two reports referenced above. When git-annex is used without `--trust-glacier`, it can end up uploading the same data multiple times. From git-annex's point of view, it cannot verify that the data is already in Glacier, so it uploads again, expecting an overwrite operation if the key is already in Glacier. Since glacier-cli maps the key to an "archive description" that can be duplicated, this is not what happens. Instead, a second archive is uploaded.

When git-annex later does a "checkpresent" operation, glacier-cli fails. This is because the request is ambiguous, since there are two archives in Glacier with the same "key". The error message could be better here, but I believe that the behaviour is correct.

#### Discussion

glacier-cli can find out what data Glacier claims to have using an inventory retrieval. However, this retrieval takes about four hours and can be out of date (eg. if someone else recently deleted the archive from another client). Thus, I can understand git-annex's desire not to trust this data or a cache of it.

However, whatever we do, it is impossible to map an "upload or overwrite on key X" type command to Glacier. We'll always end up with duplicates. Even if git-annex stored the Glacier archive IDs, there is no API to replace an existing archive with the same ID, and inventories are out of date even before we retrieve them.

#### Workaround

If the problem is as I think it is, always applying `--trust-glacier` should prevent the problem from occurring in most cases, since git-annex will run "checkpresent" and glacier-cli will confirm that the archive exists.

To fix the problem after it has occurred, it should be sufficient to delete duplicates using glacier-cli, since they _should_ be identical to each other. Some enhancement of the `glacier-cli archive list` command would help here.

Update 10 June 2013: I've pushed a `glacier-cli` update and helper script in commit `b68835`. This adds a `--force-ids` option to `glacier archive list`, with which the helper script `glacier-list-duplicates.sh` uses to identify duplicates that can be removed. If you're affected by this issue, I suggest that you use this helper to identify and fix your problem by removing the duplicates. Please do so carefully by checking that the output of the helper is correct before you use it to delete the duplicates. See the comments at the top of the helper script for usage information.

> [[fixed|done]], at least for the only well-working case for glacier, where
> only one repository can access glacier directly. --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-05-23T15:55:16Z"
 content="""
Please beware of the warning on the man page when using --trust-glacier-inventory:

> Be careful using this, especially if you or someone  else  might
> have  recently  removed  a file from Glacier. If you try to drop
> the only other copy of the file, and this switch is enabled, you
> could lose data!

While I'm inclined to want git-annex to store the necessary mappings from keys to glacier IDs in the git-annex branch, which would allow uploads/downloads from multiple repositories to the same glacier repository, it will not help with this problem. The git-annex branch can be out of date too.

It seems that what's needed is a separate form of the checkpresent hook, that's used when deciding whether to copy data to glacier.
We want this to trust the glacier inventory. But we don't want to trust the glacier inventory when moving data to glacier, or when running `git annex drop`! (unless --trust-glacier-inventory is specified). I think this would be easy to add. If you're up for testing a patch, I could do it today.

BTW, there does seem to be a workaround that avoids duplicate copies to glacier:

    git annex copy --to glacier --not --in glacier

While normally copy checks the inventory to see if a key has been sent to glacier, and so will re-send, the `--not --in glacier`
trusts the location tracking information, so if git-annex has sent the key before, it will skip the copy.
"""]]

repo_name: kakafarm/git

license: unknown


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 2"
 date="2013-05-23T15:57:08Z"
 content="""
I suppose another way to fix it along similar lines would be to make `git annex copy` always trust location tracking information when deciding whether to copy. I'm not sure how I feel about this though -- it might make things less robust in situations where `git annex copy` is run as a backup, and location tracking could have gotten out of date.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 3"
 date="2013-05-23T15:59:37Z"
 content="""
It's also worth noting that the assistant always trusts the location log when deciding whether to send a key to a remote. So I think it will not trigger this bug. It seems only `git annex copy` will. (Well, maybe `git annex move` too in an edge case.)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmUJBh1lYmvfCCiGr3yrdx-QhuLCSRnU5c"
 nickname="Justin"
 subject="comment 4"
 date="2013-05-27T22:24:44Z"
 content="""
> If you're up for testing a patch, I could do it today.

I'm happy to test a patch.  I haven't successfully compiled git-annex on my Mac, which is the only computer I have for the next month or so, but it wasn't too hard to get it to work on my Linux box.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 5"
 date="2013-05-29T17:54:11Z"
 content="""
I started to make a branch with the change I suggested, but then I had another idea.

The checkpresent hook can return either True or, False, or fail with a message if it cannot successfully check the remote. Currently for glacier, when --trust-glacier is not set, it always returns False. Crucially, in the case when a file is in glacier, this is telling git-annex it's not there, so copy re-uploads it. What if it instead, when the glacier inventory is missing a file, it returns False. And when the glacier inventory has a file, unless --trust-glacier is set, it *fails*.

The result would be:

* `git annex copy --to glacier` would only send things not listed in inventory. If a file is listed in the inventory, `copy`
  would complain that --trust-glacier` is not set, and not re-upload the file.
* `git annex drop` would only trust that glacier has a file when --trust-glacier is set. Behavior unchanged.
* `git annex move --to glacier`, when the file is not listed in inventory, would send the file, and delete it locally. Behavior unchanged.
* `git annex move --to glacier`, when the file is listed in inventory, would only trust that glacier has the file when --trust-glacier is set
* `git annex copy --from glacier` / `git annex get`, when the file is located in glacier, would trust the location log, and attempt to get the file from glacier.

This seems like it should do the right thing in all cases, but I have not tested it. I've pushed a `glacier` branch with this change.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkexhIpGcYa22aPQtLm-StpHiF-MHYPh5w"
 nickname="Robie"
 subject="comment 6"
 date="2013-06-10T17:24:34Z"
 content="""
This seems reasonable to me.

One other possibility that you could end up with a duplicate: if `glacier-cli`'s cache is not up to date. For example: hosts A and B both have (the same) annex with the same Glacier special remote defined. Host A copies a file to Glacier. On host B, the `glacier-cli` cache doesn't know about the file, and so a copy to Glacier on host B also succeeds. When the cache is later brought up to date `glacier vault sync`, then the duplicate appears.

I'm not sure what we can do about this. Perhaps we need to accept that duplicates will occur, and handle them more gracefully.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 7"
 date="2013-06-11T14:38:19Z"
 content="""
Ok, I've merged the glacier branch into master. I would still be happy to see some testing of this before my next release (in a week).

I guess I'll close this bug report. There are certainly still problems that can happen if there are multiple repositories all writing to glacier independently. Seems to me that one good way to deal with this is to set up a single remote that is configured to be a gateway to glacier.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmZgZuUhZlHpd_AbbcixY0QQiutb2I7GWY"
 nickname="Jimmy"
 subject="For those on Mac OS X"
 date="2013-11-18T00:00:32Z"
 content="""
The duplicates script fails because the BSD/MacOS version of uniq doesn't support the -D option.

You can work around this by installing the GNU version using Homebrew ('brew install coreutils') and then replacing the 'uniq' in the script with 'guniq' (Homebrew prefixes the coreutils with \"g\" by default).

I seem to still be running in to this bug using git annex version 4.20131106 and 'git annex copy --to glacier' without the '--not --in glacier' flags.  It's not a problem to use the extra flags but I wasn't originally aware of this issue and the duplicates don't seem to always occur.  I'll do some more testing and see whether I can reliably predict what will create duplicates and what won't.
"""]]


code:
Hi,

some time ago, I accidentially copied some files from the archive to the non-archive part of my (indirect, type 'client') repository (instead of moving), with the effect that assistant always kept downloading and afterwards immediately dropping the files. Now this was not really suprising once I found the duplicate folder, but maybe git annex could detect this case and refuse to run in circles or at least complain about it.

Best
Karsten

[[!tag /design/assistant]]

> Update: Current status is this is fixed for direct mode.
>
> In indirect mode, the startup scan will still download and then drop
> content if a file outside and inside the archive directory has the
> same content. It doesn't loop like it did in direct mode, only
> happens once (or once per duplicate file, really). Is still potentially
> annoying and a bug. --[[Joey]]

[[!tag confirmed]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-04-11T16:32:43Z"
 content="""
This is a known problem.

It seems possible to fix it for direct mode. After all, direct mode tracks all files associated with a key, so it could expose this to preferred content expressions, and the expression check if any of the associated files was in an archive directory.

Unsure how to deal with it in indirect mode. Short of making indirect mode do all the same tracking direct mode does, or otherwise build a key to file lookup table.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 2"
 date="2013-06-15T17:52:36Z"
 content="""
This turns out to be much worse in direct mode than in indirect mode.

In indirect mode, it only does extra work during the full startup scan. Suppose there are 3 files with the same content, 1, archive/2, and 3. It will download 1, and then will drop archive/2, and then will download 3. This certainly is not ideal, especially when the file content is large.

In indirect mode, it continally and repeatedly downloads the drops the files, as long as it's running. Which is beyond unacceptable.

What seems to be going on is that when archive/2 gets dropped, it necessary needs to convert 1 and 3 to broken symlinks. But the watcher than sees those file changes, thinks these are new or renamed files that have appeared, and promptly re-downloads them. That, in turn triggers an update of archive/2, to convert it back from symlink to direct mode file, and that in turn is noticed by the watcher. Round and round we go!
"""]]


code:
### Please describe the problem.

Direct mode repositories seem to initially ignore hard linked files and then when changes are done to them sync them as separate files. However, changes to one file are only propagated to that file and not to any of the others that are hardlinked to it.

### What steps will reproduce the problem?

Inside a direct mode repository linked to a ssh remote:

[[!format sh """
$ ls -l
total 0
$ echo "something" > foo
$ ln foo bar
$ ls -l
total 8
-rw-r--r-- 2 pedrocr pedrocr 10 May 29 12:08 bar
-rw-r--r-- 2 pedrocr pedrocr 10 May 29 12:08 foo
$ tail .git/annex/daemon.log
   6c0fbd7..0bb8ef9  git-annex -> synced/git-annex
   0bae1b4..bfedc45  master -> synced/master

sent 77 bytes  received 31 bytes  72.00 bytes/sec
total size is 10  speedup is 0.09
[2013-05-29 12:08:03 WEST] Transferrer: Uploaded foo
Already up-to-date.
[2013-05-29 12:08:05 WEST] Pusher: Syncing with golias
To ssh://golias.git-annex/home/pedrocr/testsync
   0bb8ef9..2ce5013  git-annex -> synced/git-annex
$ git status
# On branch master
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#	typechange: foo
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	bar
no changes added to commit (use "git add" and/or "git commit -a")
"""]]

On the remote repository:

[[!format sh """
$ ls -l
total 4
-rw-r--r-- 1 pedrocr pedrocr 10 May 29 12:08 foo
"""]]

If I now just touch the linked file on the repository:

[[!format sh """
$ touch bar
$ tail .git/annex/daemon.log

(merging synced/git-annex into git-annex...)
(Recording state in git...)
add bar (checksum...) [2013-05-29 12:12:49 WEST] Committer: Committing changes to git
[2013-05-29 12:12:49 WEST] Pusher: Syncing with golias
Already up-to-date.
To ssh://golias.git-annex/home/pedrocr/testsync
   2ce5013..d36166b  git-annex -> synced/git-annex
   bfedc45..ee3a7a1  master -> synced/master
Already up-to-date.
"""]]

On the remote repository:

[[!format sh """
$ ls -l
total 8
-rw-r--r-- 1 pedrocr pedrocr 10 May 29 12:08 bar
-rw-r--r-- 1 pedrocr pedrocr 10 May 29 12:08 foo
"""]]

Note that now bar has been synced as a new file and not a hardlink as it should be (the 1's after the permissions).

The sync also isn't acting properly on the linked files. For example.

First in the origin repository:

[[!format sh """
$ cat bar
something
$ cat foo
something
$ echo "someotherthing" > bar
$ cat bar
someotherthing
$ cat foo
someotherthing
"""]]

The result in the destination:

[[!format sh """
$ cat bar
someotherthing
$ cat foo
something
"""]]

So even if the intended behavior is for hardlinked files to be synced as two separate files the sync isn't correct because the two files changed in the origin and only one of them changed in the destination. This probably needs to be fixed with actual hard links for real filesystems and with some copying for crippled filesystems.

### What version of git-annex are you using? On what operating system?

[[!format sh """
$ git annex version
git-annex version: 4.20130516.1
build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP
local repository version: 4
default repository version: 3
supported repository versions: 3 4
upgrade supported from repository versions: 0 1 2
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 12.04.2 LTS
Release:	12.04
Codename:	precise
"""]]

> [[!taglink confirmed]] (but may be out of scope for git-annex) --[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkx5V3MTbzCXS3J7Mn9FEq8M9bPPYMkAHY"
 nickname="Pedro"
 subject="comment 1"
 date="2013-05-29T11:33:05Z"
 content="""
Implementing hard links will probably also require handling the edge case where the user has setup two repositories where one of them spans filesystems. So:

- Repository A is all in a single filesystem
- Repository B has it's root in one filesystem and them /someotherdirectory/ is another filesystem.

If the user in repository A does a \"ln /somefile /someotherdirectory/otherfile\" you'd need to treat this as in the crippled case as repository B can't do the hardlink spanning filesystems.

Another edge case may also be that OSX supports hardlinked directories for use with their TimeMachine feature. The feature isn't exposed through the normal ln command but users may sometimes hack around that[1]. Also, any TimeMachine backups will naturally have hardlinked directories.

[1] http://stackoverflow.com/questions/1432540/creating-directory-hard-links-in-macos-x
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 2"
 date="2013-05-29T15:39:48Z"
 content="""
git does not support hard linked files, so neither can git-annex.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 3"
 date="2013-05-29T16:11:46Z"
 content="""
It would be possible to make the assistant use the inotify CREATE event (which it currently ignores) to add a file to the repository when a hard link is created. However, when a hard linked file is modified, inotify only sends an event for the file that was changed, not for other hard links to it. So, without keeping track of all hard links that exist on my own, there's no way for the assistant to automatically handle that case. And even if it tried to, hard links to files in the repository from outside the repository would still allow modifying them without the assistant being able to detect it.

Since hard links cannot be propigated over git anyway, I don't want to get into this mess. It's best to wontfix this I think.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkx5V3MTbzCXS3J7Mn9FEq8M9bPPYMkAHY"
 nickname="Pedro"
 subject="comment 4"
 date="2013-05-29T18:48:10Z"
 content="""
I agree with your assessment for the traditional git-annex, as it's an extension of git. For the assistant (in direct mode at least) it seems like broken behavior, there will be files that exist on one side but not the other and files that have the same content on one side but not the other.

I just had a look and unfortunately there doesn't seem to be a general way to do inode to path lookup in UNIX, so an inode cache would really be needed. It would look something like:

* On inotify CREATE detect that the file has other hardlinked instances, find out if they are in the repository by keeping a inode->path cache of all files
* When the file is a hardlink to another file already in the repository commit some kind of special file to git that states \"hardlink to /path/to/otherfile\" and add metadata to the original file saying what files are hardlinks to it
* When a remote gets a special file after a sync it will itself run a \"ln /path/to/otherfile /path/to/newfile\" or alteratively a copy when the two paths span filesystems or the filesystem is crippled
* When a crippled remote gets a write to a file with hardlinks, it will replicate those writes to the other files (this is where the metadata is needed)
* When a crippled remote does a write to a file with hardlinks that file becomes independent in the remotes with hardlinks since the content is now different between the two (or more) files.

This does get pretty hairy with the corner cases. Right now the simple case of \"sync between two non-crippled filesystems\" shows pretty surprising results though so I'd argue something needs to change. I think that for direct mode to really be a \"transparent folder sync\" kind of solution this should be fixed.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.193"
 subject="comment 5"
 date="2013-06-26T16:34:23Z"
 content="""
To fix [[Problems_with_syncing_gnucash]], I have made some changes to how hard links are dealt with.

Assistant will now notice when a hard link is created, and add the same thing to git it would add for any other file. The hard link is not propigated to other repositories.

Files remain hard linked locally. This means that a change to one will affect the contents of the other. The assistant, lacking a hard link cache, will not notice this, and so will commit the change to the file that was written to, but not commit its hard link. Running `git annex add` manually (or restarting the assistant) will make it finally notice the other file has changed.

So, the assistant still does not keep hard links in sync on an ongoing basis. This bug is still unsolved.
"""]]


code:
### Please describe the problem.

[[!tag moreinfo]]

I added a lot of files to my annex in direct mode. Now I want to switch to indirect mode. git-annex status and indirect create an out-of-memory error.

### What steps will reproduce the problem?

I am not really sure, I added a lot of files to the annex, almost 3TB.
Then either git-annex status or git-annex indirect cause a similar error (see below).

### What version of git-annex are you using? On what operating system?

git-annex version: 4.20130501-g4a5bfb3
local repository version: 4
default repository version: 3
supported repository versions: 3 4
upgrade supported from repository versions: 0 1 2
build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP

Ubuntu precise
3.2.0-26-generic

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/debug.log
git-annex status
supported backends: SHA256E SHA1E SHA512E SHA224E SHA384E SHA256 SHA1 SHA512 SHA224 SHA384 WORM URL
supported remote types: git S3 bup directory rsync web webdav glacier hook
repository mode: direct
trusted repositories: 0
semitrusted repositories: 7
	00000000-0000-0000-0000-000000000001 -- web
 	0b8e6666-80d5-11e2-adf3-6f4d3d6ef0aa -- marek@x4:~/tmp/annex
 	65c057c6-6027-11e2-84b0-b77d71696e49 -- here (.)
 	96b31c5e-6524-11e2-b136-fbd1a03b2799 -- BackupOnGlacier
 	b509c388-629a-11e2-be5f-d376e201ad86 -- marek@x201:~/AllData
 	c636e33c-6e31-11e2-a9c4-a3c5546d69d9 -- desktop
 	fbaa1c3a-60d7-11e2-842f-9348368d2f4c -- .
untrusted repositories: 0
dead repositories: 0
transfers in progress: none
available local disk space: 81 gigabytes (+1 megabyte reserved)
temporary directory size: 9 megabytes (clean up with git-annex unused)
local annex keys: 61396
local annex size: 3 terabytes
known annex keys: git-annex: out of memory (requested 985661440 bytes)

OR

git-annex indirect
commit  git-annex: out of memory (requested 985661440 bytes)

# End of transcript or log.
"""]]

> [[fixed|done]]. However, if you saw this behavior,
> you have large files checked directly into git. You may
> want to examine your repository and use git filter-branch to clean
> it up.
> --[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmp1ThsNNAbSn46ju-gwFELfStlhl8usJo"
 nickname="Marek"
 subject="comment 1"
 date="2013-05-03T21:12:39Z"
 content="""
I have not yet commited after git-annex add.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 2"
 date="2013-05-06T15:59:00Z"
 content="""
Looks like you have around 61396 files, which is not really very many. Is that guess accurate?

I have repositories with that many files, that do not run out of memory, or even use much memory at all when running `git annex status`. Just tried and it needed 15 mb, not 900.

Since you're using Ubuntu precise with a current version of git-annex, I'll bet you installed from fmarier's PPA. Is that right?

In case this is a problem only with the build in the PPA, can you please download the [[/install/Linux_standalone]] tarball, and try the same command using the git-annex from there?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.14.105"
 subject="comment 3"
 date="2013-09-19T18:59:42Z"
 content="""
It turns out that if you are using direct mode and you manually `git commit some-file` that is really big, that file is checked directly into git. Which is why direct mode's docs say not to do that.

Once you've got an enormous file in git, either on purpose on by mistake, it turns out that git-annex tries to buffer the whole file content in some situations. I am in the process of fixing those memory leaks.
"""]]


code:
### Please describe the problem.
As described earlier in [[tips/fully_encrypted_git_repositories_with_gcrypt]]

### What steps will reproduce the problem?
- A: use the webapp to create a new repository
- A: add a remote server to the repository using the 'gcrypt' method
- A: add a jabber account

- B: use the webapp to create a new repository
- B: add the jabber account
- B: see the previously created 'cloud repository' with status 'not enabled'
- B: click enable, see that the stored credentials are correct, and press "verify this server"
- B: enter the ssh password twice
- B: get redirected to a blank screen (on the url /config/repository/enable/gcrypt/UUID "x"?auth=y)

The assistent logfiles show nothing after the "Your public key has been saved in", the server shows that no public key for B was added to the account.

This is with git-annex installed on the remote server; without it the process gets stuck after clicking "encrypt repository" in step 2, it will just indefinitely keep prompting for the SSH password.

### What version of git-annex are you using? On what operating system?
Latest nightly build on ubuntu 13.10

[[!tag moreinfo]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.246"
 subject="comment 1"
 date="2013-11-13T16:51:40Z"
 content="""
I've tried to reproduce this, and cannot; it enabled the repository without trouble.

Also, I have never seen the webapp fail with a blank screen, so that's strange.

I think you need to show your `~/annex/.git/annex/daemon.log` from B.
"""]]


code:
### Please describe the problem.
Incorrect merge of direct repos.

### What steps will reproduce the problem?

[[!format sh """
# setting up stuff
test/a$ git init
test/a$ git annex init
test/a$ git annex direct
test/a$ touch firstfile
test/a$ git annex add firstfile
$ git clone test/a
$ mv a test/b
test/b$ git annex direct

# actual scenario
test/b$ echo bbbb > f
test/b$ git annex add f
test/b$ git annex sync
test/a$ mkdir f
test/a$ echo aaaa > f/f
test/a$ git annex add f/f
test/a$ git annex sync
test/b$ git annex sync
test/b$ rm f
test/b$ git annex sync
test/b$ ls
test/b$ firstfile
test/b$ f.variant-SHA256E-s5--4551db5fd4d56e27be71a8a943070cfaa4342b8e960a326e2d6427b3aa0a5a48.variant-43f5
test/a$ git annex sync  # A's f/f is no longer to be found
"""]]

### What version of git-annex are you using? On what operating system?
[[!format sh """
git-annex version: 4.20131031-g7d99d14
build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav glacier hook

Linux ceilingcat 3.11.6-1-ARCH #1 SMP PREEMPT Fri Oct 18 23:22:36 CEST 2013 x86_64 GNU/Linux
"""]]

> [[fixed|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.246"
 subject="comment 1"
 date="2013-11-14T17:06:50Z"
 content="""
I verify this bug. And it's specific to direct mode as you say. Here is a shell script which automates the proccess:

[[!format sh \"\"\"
#!/bin/sh
set -e
mkdir test
cd test
git init a

cd a
git annex init
git annex direct
touch firstfile
git annex add firstfile
git annex sync # think this was left out of recipe

cd ..

git clone a b
cd b
git annex direct
echo bbbb > f
git annex add f
git annex sync || true
cd ..
cd a
mkdir f
echo aaaa > f/f
git annex add f/f
git annex sync || true
cd ..
cd b
git annex sync
echo \"after merge:\"
ls
\"\"\"]]

At this point, b only has the file version of f; the directory form has been removed. (Syncing to a of course then does the same thing there.)

And from the trascript, we can see what's going on:

<pre>
Adding f/f
CONFLICT (directory/file): There is a directory with name f in HEAD. Adding f as f~refs_heads_synced_master
Automatic merge failed; fix conflicts and then commit the result.
(Recording state in git...)
f: needs merge
[master 0600854] git-annex automatic merge conflict fix

  Merge conflict was automatically resolved; you may want to examine the result.
</pre>

The problem seems to be that direct mode merge does not find the `f~refs_heads_synced_master` created by the merge, so fails to copy it from the temp merge tree into the work tree.

`Command.Sync.cleanConflictCruft` is relevant, but was only made to work in indirect mode, it seems.

----

Obviously, if someone runs into this bug and seems to lose data, they can get the data back by reverting the changes from the automatic merge. Direct mode does preserve file contents when removing them from the work tree in a merge.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.246"
 subject="comment 2"
 date="2013-11-15T17:39:37Z"
 content="""
My initial guess was wrong.. This is not actually a bug in conflicted merge resolution at all.

The bug is that in direct mode, it diffs the old and new tree when doing a normal merge, so see what files in the work tree need to be changed. This was written to go through the diff and replay the deletes and adds. In this case, since f/f and f are different items, they can appear in either order in the diff But the code only worked when f was first deleted, and f/f was then added. And it turns out that in this case, the diff had the two items the other way around.

So, I think it needs to do 2 passes, first deleting and then adding.
"""]]


code:
### Please describe the problem.
(Minor issue.)

Incorrect merge of direct repos in the special case where at repo A a symlink to a file whose contents aren't yet available, are overwritten, while at repo B the file is deleted.

Result: file is deleted on both side.

Expected: B.f is gone, A.f is still present

### What steps will reproduce the problem?

[[!format sh """
# setting up stuff
test/a$ git init
test/a$ git annex init
test/a$ git annex direct
test/a$ touch firstfile
test/a$ git annex add firstfile
$ git clone test/a
$ mv a test/b
test/b$ git annex direct

# actual scenario
test/b$ echo bbbb > f
test/b$ git annex add f
test/b$ git annex sync
test/a$ git annex sync
test/a$ echo aaaa > f
test/a$ git annex add f
test/a$ git annex sync
test/b$ rm f
test/b$ git annex sync
test/a$ git annex sync
# test/a/f is now gone, lost
"""]]

### What version of git-annex are you using? On what operating system?
[[!format sh """
git-annex version: 4.20131031-g7d99d14
build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav glacier hook

Linux ceilingcat 3.11.6-1-ARCH #1 SMP PREEMPT Fri Oct 18 23:22:36 CEST 2013 x86_64 GNU/Linux
"""]]

> [[fixed|done]]; direct mode now freezes the content directory as indirect
> mode already did. fsck will fix up the permissions too. --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.246"
 subject="comment 1"
 date="2013-11-14T17:10:55Z"
 content="""
I suspect this might be the same underlying problem as [[bugs/Incorrect merge, direct repos (2)]]. However, I cannot reproduce it using the recipe given.. perhaps something was left out?

I wrote this shell script to try to codify the recipe in a runnable form:

[[!format sh \"\"\"
#!/bin/sh
set -e
mkdir test
cd test
git init a

cd a
git annex init
git annex direct
touch firstfile
git annex add firstfile
git annex sync # think this was left out of recipe

cd ..

git clone a b
cd b
git annex direct
echo bbbb > f
git annex add f
git annex sync || true
cd ..
cd a
echo aaaa > f
git annex add f
git annex sync
\"\"\"]]

At this point, a has 2 variants of f, and no amount of syncing in either repo will cause either variant to go away.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlWskoNgUB7r70OXglR-4iKI4bOuPJb-xg"
 nickname="Tim"
 subject="comment 2"
 date="2013-11-14T18:06:55Z"
 content="""
You were missing a: test/a$ git annex sync

This did the trick on my system
[[!format sh \"\"\"
#!/bin/sh
set -e
mkdir test
cd test
git init a

cd a
git annex init
git annex direct
touch firstfile
git annex add firstfile
git annex sync # think this was left out of recipe  # indeed it was

cd ..

git clone a b
cd b
git annex direct
echo bbbb > f
git annex add f
git annex sync || true  # why add a || true?
cd ../a
git annex sync
echo aaaa > f
git annex add f
git annex sync
cd ../b
rm f
git annex sync
ls
cd ../a
git annex sync
ls
\"\"\"]]
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.246"
 subject="comment 3"
 date="2013-11-15T17:48:42Z"
 content="""
Hmm. In your script, when you run `git annex sync` in a and then `echo aaaa > f`, f already exists at that point as a symlink. This actually ends up following the link and writing to .git/annex/objects. (fsck will detect that junk has been written there)

That's a bug on its own; seems like direct mode is neglecting to lock down the .git/annex/objects directories to prevent writing to them like this.

----

However, this means that your script does not demonstrate the bug you originally reported.
You remove b/f and sync, and since a/f has not been changed, the deleting is correctly synced to a, removing a/f.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.246"
 subject="comment 4"
 date="2013-11-15T17:52:38Z"
 content="""
Looking back at the original bug description:

\"repo A a symlink to a file whose contents aren't yet available, are overwritten, while at repo B the file is deleted.\"

I think the \"overwritten\" is key. I suspect you were always doing echo > f where f was a symlink, and this does not actually overwrite the symlink, it just puts data (that fsck will reject) into the annex.

So, proceeding as if the real bug here is the ability to write through symlink in direct mode, unless told otherwise..
"""]]


code:
### Please describe the problem.

Git Annex is listed as of version 1.0.52 if you check out its entry in the list of applications in the Android configuration.  I guess that one is of the terminal app which kickstarts the annex.

### What steps will reproduce the problem?

install apk

### What version of git-annex are you using? On what operating system?

android
5.2013whatever

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

# End of transcript or log.
"""]]

> [[fixed|done]] --[[Joey]]


code:
### Please describe the problem.

[380 of 462] Compiling Assistant.WebApp.Types ( Assistant/WebApp/Types.hs, dist/build/git-annex/git-annex-tmp/Assistant/WebApp/Types.o )

Assistant/WebApp/Types.hs:157:10:
    Duplicate instance declarations:
      instance PathPiece Bool
        -- Defined at Assistant/WebApp/Types.hs:157:10
      instance PathPiece Bool
        -- Defined in `path-pieces-0.1.4:Web.PathPieces'
cabal: Error: some packages failed to install:
git-annex-5.20140709 failed during the building phase. The exception was:
ExitFailure 1

### What steps will reproduce the problem?

cabal install git-annex --bindir=$HOME/bin

### What version of git-annex are you using? On what operating system?
git-annex-5.20140709, Fedora 20

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

# End of transcript or log.
"""]]

> Already fixed in git yesterday. [[done]] --[[Joey]]


code:
### Please describe the problem.

One of my repositories has no name:
http://screencast.com/t/3OjxFzpz

And when I try to disable it I get this error:

    Internal Server Error
    Unknown UUID

When I try to delete it I get this error:

    Internal Server Error
    unknown UUID; cannot modify

I think this was the result of adding a Local Computer Repo, and then that computer signed off.  Maybe.

### What version of git-annex are you using? On what operating system?

git-annex version 4.20130601-g2b6c3f2
Mac OS 10.7.5

### Please provide any additional information below.

Maybe it's a glitch that only will happen this once, the problem is I can't get rid of it!  Are there anyways of manually getting rid of a repo with uid?

> Also reported here:
> [[Missing_repo_uuid_after_local_pairing_with_older_annex]] and
> [[Internal_Server_Error_unknown_UUID;_cannot_modify]]
> and [[Local_network___40__ssh__41___fails_to_pair__47__sync]]
> and [[Internal_Server_Error:_Unknown_UUID]]
> --[[Joey]]

[[!meta title="local pairing leads to unknown UUID"]]

> This bug is [[fixed|done]]. The webapp will detect the problem and
> provides an interface to correct it. --[[Joey]]


code:
[[!comment format=mdwn
 username="carlo"
 ip="118.208.1.126"
 subject="comment 1"
 date="2013-07-06T22:52:18Z"
 content="""
I got the same error. Local repo has no name but in the logs I can see that it's trying to make a local connection:

    [2013-07-07 08:29:51 EST] main: starting assistant version 4.20130627
    [2013-07-07 08:29:51 EST] TransferScanner: Syncing with arkham.local_annex

(snip)

    07/Jul/2013:08:41:56 +1000 [Error#yesod-core] Unknown UUID @(yesod-core-1.2.3:Yesod.Core.Class.Yesod ./Yesod/Core/Class/Yesod.hs:471:5)
    ssh: Could not resolve hostname arkham.local: Name or service not known
    ssh: Could not resolve hostname arkham.local: Name or service not known
    fatal: The remote end hung up unexpectedly
    git-annex: Unknown UUID
    ssh: Could not resolve hostname arkham.local: Name or service not known
    ssh: Could not resolve hostname arkham.local: Name or service not known
    fatal: The remote end hung up unexpectedly
    git-annex: Unknown UUID
    ssh: Could not resolve hostname arkham.local: Name or service not known
    ssh: Could not resolve hostname arkham.local: Name or service not known
    fatal: The remote end hung up unexpectedly
    git-annex: Unknown UUID
    ssh: Could not resolve hostname arkham.local: Name or service not known
    07/Jul/2013:08:42:32 +1000 [Error#yesod-core] Unknown UUID @(yesod-core-1.2.3:Yesod.Core.Class.Yesod ./Yesod/Core/Class/Yesod.hs:471:5)
    ssh: Could not resolve hostname arkham.local: Name or service not known
    fatal: The remote end hung up unexpectedly
    git-annex: Unknown UUID
    ssh: Could not resolve hostname arkham.local: Name or service not known

When I turn on arkham (the other laptop) I see transfers start up:

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkeyfC88gIU4fsEqqkvhzVlSKDUxbtZaTE"
 nickname="Selem"
 subject="comment 2"
 date="2013-07-10T01:41:24Z"
 content="""
I got the same problem (Mac OS X 10.8.4, whatever the latest download package, local computers same network repos)

To get rid of the noname repo, I just removed git remote in the annex dir.

    # List remote
    $ git remote -v
    # Remove remote grabbed from running the previous command
    $ git remote rm your_remote

After that, I added the second computer repo successfully.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.246.110"
 subject="I can't reproduce this problem, but I see several people have reported it."
 date="2013-07-27T22:11:15Z"
 content="""
It would be helpful for anyone who can reproduce this problem to do so with debugging enabled on both sides
 (run with --debug or turn it on in the Configuration page in webapp), and then send the `.git/annex/daemon.log`
"""]]


code:
[[!comment format=mdwn
 username="https://launchpad.net/~subito"
 nickname="subito"
 subject="Same here for SSH remotes"
 date="2013-07-29T19:05:55Z"
 content="""
I get the problem while only adding an SSH remote. After deciding I want a git-repo on my server, it says \"Unknown UUID\" and created a remote with no name. I was unable to add any SSH remote (tried with two different servers and a couple different dirs - some completly unknown to my annex)

I used the Android webapp btw - latest Version. The same version works fine on my Debian maschines.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.21"
 subject="comment 5"
 date="2013-07-30T16:59:50Z"
 content="""
@subito I have tried as hard as I can to reproduce this problem, and cannot. I need the debug information I asked for in my other comment to get any further on this bug report.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.21"
 subject="finally!"
 date="2013-07-31T16:47:39Z"
 content="""
I've finally been clued in to the cause of this. If a ssh-agent is running, and has been configured to use a ssh key (by running `ssh-add`), this apparently prevents git-annex from using the key it sets up during local pairing. I have reproduced the described bug by this method.

Something similar might also affect adding a remote ssh server, I'm not sure about that yet.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.21"
 subject="workaround"
 date="2013-07-31T17:19:01Z"
 content="""
Looks like the fix is to edit ~/.ssh/config and in each stanza added by git-annex, add this line:

        IdentitiesOnly yes

This prevents the ssh-agent from using the normal key, and allows the git-annex key to be used instead.

People experiencing this bug can manually make that change. Then if you edit your git-annex repository's `.git/config` and delete the `annex-ignore` line, the assistant should finally learn the UUID and start syncing.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.21"
 subject="comment 8"
 date="2013-07-31T20:42:10Z"
 content="""
I've updated the webapp to display nicely when a repository has stalled being set up, rather than crashing.

There's now an interface to check the status of such a stalled repository, and it will try to clean up after this bug too.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnZEanlyzay_QlEAL0CWpyZcRTyN7vay8U"
 nickname="Carlo"
 subject="Tilde did it for me"
 date="2013-10-06T21:59:43Z"
 content="""
I had exactly the same error message, \"Internal Server Error: Unknown UUID\", when I used a path starting with a tilde. Absolute path for home directory worked.
"""]]


code:
### Please describe the problem.
I was trying to use "Local Computer" option to sync up two machines on my local network. I was having some firewall issues and it failed on one machine.
It still created a repository without a name in the web ui and i get that Error unknown UUID when trying to edit or delete it.

### What steps will reproduce the problem?
Machine A initiates pairing. Machine B accepts pairing. Machine A has a firewall blocking outgoing connections.
Machine B times out. Machine A accepts outgoing connections for vnetd. Machine A starts syncing with Machine B.
Machine B gets files but have a broken web ui.

### What version of git-annex are you using? On what operating system?

4.20130601-g2b6c3f2
OSX 10.7.5 on both Machine A and B

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

# End of transcript or log.
"""]]

> [[dup|done]] of [[Internal_Server_Error:_Unknown_UUID]] --[[Joey]]


code:
What steps will reproduce the problem?

Adding files to a local annex set up to sync to a remote S3 one

What is the expected output? What do you see instead?

It syncs, but maxes out the uplink

What version of git-annex are you using? On what operating system?

3.20121112 on Debian testing

Please provide any additional information below.

The man page lists how to configure rate limiting for rsync, not sure how to do it for this

[[!tag confirmed]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo"
 nickname="Justin"
 subject="comment 1"
 date="2012-11-13T15:10:45Z"
 content="""
There's always trickle..

    trickle -u 50 git annex ...
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.252.11.120"
 subject="comment 2"
 date="2012-11-13T17:09:31Z"
 content="""
It's not there yet. I don't think it will be hard to add, most of the same things need to be done to support upload progress bars with S3.
"""]]


code:
I was dumping ~gigs of files of approximately 3-6megs a pop (my music collection) so I could track the files that I want to listen to when I'm on the go. I had the git watch command running from the assistant branch.

I was getting something along the lines of...

    /Users/jtang/annex/.git/annex/tmp/: openTempFile: resource exhausted (Too many open files)

and

    git-annex: createPipe: resource exhausted (Too many open files)

I also noticed that I somehow ended up with 256 ssh-agent's running on one of my machines, I'm not sure if the two issues are related or not, I had not noticed this type of behaviour up until recently.

Also this was appearing in the logs

    x00:annex jtang$ tail -f .git/annex/daemon.log
    (scanning...) Already up-to-date.
    kqueue: Too many open files

To be precise, I suspect that the kqueue limit is 256, I had 325 files in the 'queue', I ended up doing a _git annex add_ manually and all was fine.

[[!meta title="kqueue system limits"]]

> This affects BSD systems that use Kqueue. It no longer affects OSX,
> since we use FSEvents there instead. --[[Joey]]

[[!tag /design/assistant]]
[[!tag confirmed]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.2.25"
 subject="comment 1"
 date="2012-06-25T15:42:48Z"
 content="""
Yes, this is a known problem with kqueue, it has to keep every directory in the tree open. On [[design/assistant/inotify]] I have a note that it may need to fork off extra watcher processes to deal with this. Of course that adds significant complication.

In the meantime, you may be able to increase your system's maximum allowed number of open files per process somehow.

(I doubt that the ssh-agent is related; git-annex does not use ssh-agent directly anyway..)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 2"
 date="2012-06-29T12:02:48Z"
 content="""
Doing,

    sudo sysctl -w kern.maxfilesperproc=400000

Somewhat works for me, git-annex watch at least starts up and takes a while to scan the directory, but it's not ideal. Also, creating files seems to work okay, when I remove a file the changes don't seem to get pushed across my other repos, running a sync on the remote repo fixes things.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 subject="comment 3"
 date="2012-07-04T12:32:44Z"
 content="""
Jimmy, sounds like I could use something like this to get the current limit:

    sysctl kern.maxfilesperproc

Probably prints \"sysctl kern.maxfilesperproc = 256\" or such.. can you verify?
Once I have the limit, I can make the kqueue code use subset of it, and print out a message when it needs to be increased, like the inotify code does.

(Also, the kqueue code only opens directories, not files, so unless you have 400000 directories, that's
a little high.)

---

On file removal not propigating, does this still happen? When you remove a file does a git commit automatically happen, or is that broken with kqueue?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 subject="comment 4"
 date="2012-07-08T18:13:58Z"
 content="""
On kFreeBSD, I get this:

    $ sysctl kern.maxfilesperproc
    kern.maxfilesperproc: 11095

But ulimit still has 1024 limit, so you'd need to adjust both, as root. Messy..
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 4"
 date="2012-07-04T13:17:05Z"
 content="""
In relation to the system limits,

    laplace:~ jtang$ sysctl kern.maxfilesperproc
    kern.maxfilesperproc: 10240

Also, the maxfiles for the whole system is

    laplace:~ jtang$ sysctl kern.maxfiles
    kern.maxfiles: 12288

the above was the defaults as far as I recall. What you probably would be interested is the ulimits that the user see

    laplace:~ jtang$ ulimit -a
    core file size          (blocks, -c) 0
    data seg size           (kbytes, -d) unlimited
    file size               (blocks, -f) unlimited
    max locked memory       (kbytes, -l) unlimited
    max memory size         (kbytes, -m) unlimited
    open files                      (-n) 256
    pipe size            (512 bytes, -p) 1
    stack size              (kbytes, -s) 8192
    cpu time               (seconds, -t) unlimited
    max user processes              (-u) 709
    virtual memory          (kbytes, -v) unlimited

I would imagine the limit that you are looking for is 256. Hope this helps.

----

On the point about deletions not being propagated, it does do a commit. I suspect that the kqueue code is just not picking up the changes and pushing the changes out. The watch command on a single annex with no remotes functions as expected.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 2"
 date="2012-06-25T22:36:39Z"
 content="""
On the system limits side, I think if you want to make it more approachable by more users then adjusting system limits might scare users away. On the note of the ssh-agents spawning like no tomorrow on my machine, it turned out that i had a symlink from my .bashrc to .bash_profile, I guess I should not be too lazy and have two seperate files.
"""]]


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="173.162.44.162"
 subject="Just encountered this today"
 date="2012-11-29T19:21:50Z"
 content="""
& it's killing my assistant.

Found a workaround here: https://github.com/mockko/livereload/issues/1

This fixes it for now:

ulimit -n 4096

Just posting here in case anybody else runs into this issue and, like me, finds themselves here via search engine.

"""]]


code:
What steps will reproduce the problem?

Run the git-annex assitant, and then "sudo kill" it.

What is the expected output? What do you see instead?

I expect it to die, instead I end up with:

    14604   ??  S      0:00.64 ga assistant
    14623   ??  Z      0:00.00 (git)
    14624   ??  Z      0:00.00 (git)
    14936   ??  Z      0:00.00 (git-annex)

The only way to clear these zombies is to reboot.  Perhaps there is some resource not being correctly terminated under exceptional conditions?

Note that on OpenIndiana the problem is even more severe: Aborting git-annex at the wrong time leaves behind both zombie processes and lock files which cause the machine to suddenly halt if I try to access them in any way (via mv, rsync, etc)!

What version of git-annex are you using? On what operating system?

4d1e0c9 on OS X 10.8.2.

Please provide any additional information below.

[[!meta title="strange OSX behavior when killed"]]
[[!tag /design/assistant/OSX moreinfo]]


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/2grhJvAC049fJnvALDXek.6MRZMTlg--#eec89"
 nickname="John"
 subject="Correction"
 date="2012-10-20T05:34:47Z"
 content="""
If I \"sudo kill -9\" all the processes in the above group after deleting the annex directories, then they do go away without needing to reboot.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.23"
 subject="comment 2"
 date="2012-10-22T14:01:42Z"
 content="""
It's not at all clear to me from this what process you killed.

(I can't comment on files that somehow crash a kernel on access.. That's not in any UNIX spec I'm aware of.)
"""]]


code:
    [0 zerodogg@firefly git-annex]$ ls
    bin/  favicon.png  git-annex           git-annex-shell   git-core/                 lib/    libdirs  linker    README    shimmed/    uninstall
    etc/  gconvdir     git-annex.MANIFEST  git-annex-webapp  install-haskell-packages  lib64/  LICENSE  logo.svg  runshell  templates/  usr/
    [0 zerodogg@firefly git-annex]$ pwd
    /home/zerodogg/.local/share/git-annex
    [0 zerodogg@firefly git-annex]$ ./git-annex version
    /home/zerodogg/.local/share/git-annex/bin/git-annex: line 2: /home/zerodogg/.local/share/git-annex/lib64/ld-linux-x86-64.so.2: No such file or directory
    [127 zerodogg@firefly git-annex]$ ls -l /home/zerodogg/.local/share/git-annex/lib64/ld-linux-x86-64.so.2
    lrwxrwxrwx. 1 zerodogg zerodogg 32 2012-12-30 16:35 /home/zerodogg/.local/share/git-annex/lib64/ld-linux-x86-64.so.2 -> /lib/x86_64-linux-gnu/ld-2.13.so
    [0 zerodogg@firefly git-annex]$ ls /lib/x86_64-linux-gnu/ld-2.13.so
    ls: cannot access /lib/x86_64-linux-gnu/ld-2.13.so: No such file or directory

This is on Fedora 20, 64bit, installed from http://downloads.kitenet.net/git-annex/linux/current/git-annex-standalone-amd64.tar.gz as of Sun Dec 22 00:44:53 CET 2013.

> [[fixed|done]]; converted ugly shell code to much nicer haskell code,
> which let me reuse relative symlink generation code from git-annex
> to fix this. Updated the amd64 build for the last release only.
> --[[Joey]]


code:
[[!comment format=mdwn
 username="EskildHustvedt"
 ip="80.202.194.17"
 subject="32 bit standalone works (on 64 bit) however"
 date="2013-12-22T11:37:24Z"
 content="""
SSIA
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmTlfbCC37CAjhQrS107ZWRVA_sF4s3gLU"
 nickname="Juergen"
 subject="Same here  broken "
 date="2013-12-22T12:11:57Z"
 content="""
It's broken here too. Debian testing 64bit .

"""]]


code:
[[!comment format=mdwn
 username="http://id.clacke.se/"
 nickname="clacke"
 subject="Workaround/Fix"
 date="2013-12-22T14:23:09Z"
 content="""
Broken here on Linux Mint 13 Maya (Ubuntu 12.04 precise) too. The reason is a dangling symlink. Fix:

    clacke@acozed:~/.local/libexec/git-annex.linux.5.20131221$ ./git-annex version
    Segmentation fault
    clacke@acozed:~/.local/libexec/git-annex.linux.5.20131221$ rm lib64/ld-linux-x86-64.so.2
    clacke@acozed:~/.local/libexec/git-annex.linux.5.20131221$ ln -s ../lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 lib64/
    clacke@acozed:~/.local/libexec/git-annex.linux.5.20131221$ ./git-annex version
    git-annex version: 5.20131221-gf8c928d
    build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus XMPP Feeds Quvi TDFA
    key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SHA256 SHA1 SHA512 SHA224 SHA384 WORM URL
    remote types: git gcrypt S3 bup directory rsync web webdav glacier hook
"""]]


code:
[[!comment format=mdwn
 username="http://grossmeier.net/"
 nickname="greg"
 subject="comment 4"
 date="2013-12-23T16:45:23Z"
 content="""
clacke's workaround works for me, as well, on 5.20131223-gafb776f (ie: issue still present on 5.20131223-gafb776f)
"""]]


code:
### Please describe the problem.
git-annex: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found (required by git-annex)
git-annex: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found (required by /home/user/bin/usr/lib/x86_64-linux-gnu/libidn.so.11)
git-annex: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found (required by /home/user/bin/usr/lib/x86_64-linux-gnu/libgnutls.so.26)
git-annex: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found (required by /home/user/bin/usr/lib/x86_64-linux-gnu/libxml2.so.2)
git-annex: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.15' not found (required by /home/user/bin/usr/lib/x86_64-linux-gnu/libxml2.so.2)
git-annex: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found (required by /home/user/bin/usr/lib/x86_64-linux-gnu/libgmp.so.10)
git-annex: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found (required by /home/user/bin/usr/lib/x86_64-linux-gnu/libffi.so.6)
git-annex: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found (required by /home/user/bin/usr/lib/x86_64-linux-gnu/libntlm.so.0)
git-annex: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found (required by /home/user/bin/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2)
git-annex: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found (required by /home/user/bin/usr/lib/x86_64-linux-gnu/libkrb5.so.3)
git-annex: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.16' not found (required by /home/user/bin/usr/lib/x86_64-linux-gnu/libkrb5.so.3)
git-annex: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found (required by /home/user/bin/usr/lib/x86_64-linux-gnu/libk5crypto.so.3)
git-annex: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.17' not found (required by /home/user/bin/lib/x86_64-linux-gnu/libcom_err.so.2)
git-annex: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found (required by /home/user/bin/lib/x86_64-linux-gnu/libgcrypt.so.11)
git-annex: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.15' not found (required by /home/user/bin/lib/x86_64-linux-gnu/libgcrypt.so.11)
git-annex: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found (required by /home/user/bin/usr/lib/x86_64-linux-gnu/libtasn1.so.3)
git-annex: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found (required by /home/user/bin/usr/lib/x86_64-linux-gnu/libp11-kit.so.0)
git-annex: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found (required by /home/user/bin/usr/lib/x86_64-linux-gnu/libstdc++.so.6)
git-annex: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.17' not found (required by /home/user/bin/usr/lib/x86_64-linux-gnu/libstdc++.so.6)
git-annex: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found (required by /home/user/bin/lib/x86_64-linux-gnu/libgcc_s.so.1)
git-annex: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found (required by /home/user/bin/usr/lib/x86_64-linux-gnu/libkrb5support.so.0)

### What steps will reproduce the problem?
$git-annex

### What version of git-annex are you using? On what operating system?
v20130723, 64-bit, Debian Stable

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

# End of transcript or log.
"""]]

Also note, the last build 20130709 was compatible with glibc 2.13

> [[fixed|done]]; builds are back to using debian stable.
> also updated the autobuilds and last release's builds. --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4978:f:21a::2"
 subject="comment 1"
 date="2013-07-25T17:00:53Z"
 content="""
Thank you for reminding me why I was using debian stable chroots for the autobuilds.

Unfortunately, I forgot about this problem and upgraded the chroots. Rebuilding them with all the stuff I need is going to take a while.
"""]]


code:
[[!comment format=mdwn
 username="http://jamdev.myopenid.com/"
 ip="117.195.168.15"
 subject="well that was fast!"
 date="2013-07-25T21:01:38Z"
 content="""
Hey Joey

Confirming this as fixed, thanks.

-have a nice weekend!
"""]]


code:
### Please describe the problem.
I am trying to set out two computers on the same network to synchronise.

### What steps will reproduce the problem?
Install Git-Annex. Start the webapp. Try to connect. Enter a secret phrase on both. The Mythbuntu machine shows "Failed to sync with Inspiron 14z" (the laptop). The laptop shows "Pairing in progress" forever.

The machines can normally connect together passwordlessly through ssh with public key encryption.

### What version of git-annex are you using? On what operating system?
Ubuntu Raring Version: 3.20121112ubuntu2 from the repos on my laptop.  Install version 4.20130627 from the PPA on Mythbuntu Precise (which I use as a home server).

### Please provide any additional information below.
From the the laptop, where I started the pairing:
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
$ git-annex webapp

(process:3084): GLib-CRITICAL **: g_slice_set_config: assertion `sys_page_size == 0' failed

** (firefox:3084): WARNING **: Failed to find domain member of JSON manifest
Running global cleanup code from study base classes.
(Recording state in git...)

Launching web browser on file:///tmp/webapp3075.html
(scanning...)
  dbus failed; falling back to mtab polling (ClientError {clientErrorMessage = "Call failed: The name org.gtk.Private.GduVolumeMonitor was not provided by any .service files", clientErrorFatal = False})
(started...) Generating public/private rsa key pair.
Your identification has been saved in /tmp/git-annex-keygen3075.0/key.
Your public key has been saved in /tmp/git-annex-keygen3075.0/key.pub.
The key fingerprint is:
79:00:67:a4:f0:5f:62:26:78:ed:09:97:e4:c4:dd:56 aaron@Inspiron-14z
The key's randomart image is:
+--[ RSA 2048]----+
|    . .o*. . .E  |
|     + X... o    |
|    . * X ..     |
|     . O *       |
|        S .      |
|         .       |
|                 |
|                 |
|                 |
+-----------------+
Control socket connect(/home/shared/annex/.git/annex/ssh/mythbuntu@git-annex-mythbuntu-server.local-mythbuntu): Connection refused
Failed to connect to new control master
warning: no common commits

  Remote mythbuntuserver.local_annex does not have git-annex installed; setting remote.mythbuntuserver.local_annex.annex-ignore
Already up-to-date.
Counting objects: 13, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (9/9), done.
Writing objects: 100% (11/11), 1.06 KiB, done.
Total 11 (delta 2), reused 0 (delta 0)
To ssh://mythbuntu@git-annex-mythbuntu-server.local-mythbuntu/~/annex/
 * [new branch]      git-annex -> synced/git-annex
 * [new branch]      master -> synced/master
Already up-to-date!
Merge made by the 'recursive' strategy.
Already up-to-date.
Already up-to-date.
Counting objects: 2, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 326 bytes, done.
Total 2 (delta 1), reused 0 (delta 0)
To ssh://mythbuntu@git-annex-mythbuntu-server.local-mythbuntu/~/annex/
   82bf946..dfe0bd1  master -> synced/master
Already up-to-date.
SendMessage (77594660, 0x101f, (nil), (nil))
SendMessage (0, 0x1203, (nil), 0x7fffc68a1850)
SendMessage (0, 0x1204, (nil), 0x7fffc68a1850)
SendMessage (0, 0x1203, 0x1, 0x7fffc68a1850)
SendMessage (0, 0x1204, 0x1, 0x7fffc68a1850)
SendMessage (0, 0x1203, 0x2, 0x7fffc68a1850)
SendMessage (0, 0x1204, 0x2, 0x7fffc68a1850)
SendMessage (0, 0x1203, 0x3, 0x7fffc68a1850)
SendMessage (0, 0x1204, 0x3, 0x7fffc68a1850)
SendMessage (0, 0x1203, 0x4, 0x7fffc68a1850)
SendMessage (0, 0x1204, 0x4, 0x7fffc68a1850)
SendMessage (77594660, 0x101f, (nil), (nil))
SendMessage (0, 0x1203, (nil), 0x7fffc68a1810)
SendMessage (0, 0x1204, (nil), 0x7fffc68a1810)
SendMessage (0, 0x1203, 0x1, 0x7fffc68a1810)
SendMessage (0, 0x1204, 0x1, 0x7fffc68a1810)
SendMessage (0, 0x1203, 0x2, 0x7fffc68a1810)
SendMessage (0, 0x1204, 0x2, 0x7fffc68a1810)
SendMessage (0, 0x1203, 0x3, 0x7fffc68a1810)
SendMessage (0, 0x1204, 0x3, 0x7fffc68a1810)
SendMessage (0, 0x1203, 0x4, 0x7fffc68a1810)
SendMessage (0, 0x1204, 0x4, 0x7fffc68a1810)
SendMessage (77594660, 0x101f, (nil), (nil))
SendMessage (0, 0x1203, (nil), 0x7fffc68a2920)
SendMessage (0, 0x1204, (nil), 0x7fffc68a2920)
SendMessage (0, 0x1203, 0x1, 0x7fffc68a2920)
SendMessage (0, 0x1204, 0x1, 0x7fffc68a2920)
SendMessage (0, 0x1203, 0x2, 0x7fffc68a2920)
SendMessage (0, 0x1204, 0x2, 0x7fffc68a2920)
SendMessage (0, 0x1203, 0x3, 0x7fffc68a2920)
SendMessage (0, 0x1204, 0x3, 0x7fffc68a2920)
SendMessage (0, 0x1203, 0x4, 0x7fffc68a2920)
SendMessage (0, 0x1204, 0x4, 0x7fffc68a2920)
Redirection loop trying to set HTTPS on:
  http://www.aol.com/favicon.ico
(falling back to HTTP)
SendMessage (77594652, 0x444, 0x1, 0x3652e00)
SendMessage (77594652, 0x444, 0x1, 0x3647de0)
SendMessage (77594652, 0x444, 0x1, 0x3667a80)
SendMessage (77594652, 0x444, 0x1, 0x3667a80)

# End of transcript or log.
# End of transcript or log.
"""]]

On the Mythbuntu machine:
[[!format sh """
$ git-annex webapp
Launching web browser on file:///tmp/webapp8399.html
(Recording state in git...)
"""]]

Unless I'm going mad, there doesn't seem to be a daemon.log on my laptop.

Daemon.log on the Mythbuntu machine:
[[!format sh """
[2013-07-14 10:38:56 BST] main: starting assistant version 4.20130627
(scanning...) [2013-07-14 10:38:56 BST] Watcher: Performing startup scan
(started...) [2013-07-14 10:38:57 BST] PairListener: aaron@Inspiron-14z:/home/shared/annex is sending a pair request.
Generating public/private rsa key pair.
Your identification has been saved in /tmp/git-annex-keygen.0/key.
Your public key has been saved in /tmp/git-annex-keygen.0/key.pub.
The key fingerprint is:
bb:8c:66:05:22:8e:fa:e1:10:33:6d:cb:d6:57:e2:47 mythbuntu@mythbuntu-server
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                 |
|                 |
| .. . .          |
|+oo. ...E        |
|.*.o . +..       |
|o = . o.o        |
|.+ . .o+ .       |
| .o  o. o        |
+-----------------+
[2013-07-14 10:39:13 BST] main: Pairing with aaron@Inspiron-14z:/home/shared/annex in progress
ssh: connect to host Inspiron-14z.local port 22: Connection refused
ssh: connect to host Inspiron-14z.local port 22: Connection refused
fatal: The remote end hung up unexpectedly
[2013-07-14 10:39:17 BST] PairListener: Syncing with Inspiron14z.local__home_shared_annex
ssh: connect to host Inspiron-14z.local port 22: Connection refused
fatal: The remote end hung up unexpectedly
ssh: connect to host Inspiron-14z.local port 22: Connection refused
fatal: The remote end hung up unexpectedly
ssh: connect to host Inspiron-14z.local port 22: Connection refused
fatal: The remote end hung up unexpectedly
Already up-to-date.
Already up-to-date.
ssh: connect to host Inspiron-14z.local port 22: Connection refused
fatal: The remote end hung up unexpectedly
ssh: connect to host Inspiron-14z.local port 22: Connection refused
fatal: The remote end hung up unexpectedly
Updating 82bf946..dfe0bd1
Fast-forward
[2013-07-14 10:39:56 BST] Pusher: Syncing with Inspiron14z.local__home_shared_annex
ssh: connect to host Inspiron-14z.local port 22: Connection refused
fatal: The remote end hung up unexpectedly
ssh: connect to host Inspiron-14z.local port 22: Connection refused
fatal: The remote end hung up unexpectedly
ssh: connect to host Inspiron-14z.local port 22: Connection refused
fatal: The remote end hung up unexpectedly
ssh: connect to host Inspiron-14z.local port 22: Connection refused
fatal: The remote end hung up unexpectedly
"""]]

[[!taglink moreinfo]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnFjuvfPpi1kf6l54bxfFUm0Aw_Gf_IO0o"
 nickname="Aaron"
 subject="Some things working"
 date="2013-07-14T10:06:13Z"
 content="""
Oh, that's odd...

The Mythbuntu machine says it fails and the laptop screen still says pairing, but a new window on the laptop says that it synced with the Mythbuntu server. On the Mythbuntu server, it has a blank entry for the laptop, which if you click to edit says \"Internal Server Error\" \"Unknown UUID\".

When I add things on the laptop, they immediately become shortcuts and an equivalent shortcut appears on the Mythbuntu machine, though updates to the content (in this case a text file) are not sychronised.

Adding a file on the Mythbuntu machine does not turn into a shortcut and nothing appears on the laptop.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.246.110"
 subject="comment 2"
 date="2013-07-28T00:27:23Z"
 content="""
Remote mythbuntuserver.local_annex does not have git-annex installed; setting remote.mythbuntuserver.local_annex.annex-ignore

So, how did you install git-annex on mythbuntuserver? What does the `.ssh/authorized_keys` on mythbuntuserver look like? (In particular the key that git-annex adds to it, which should be set to run git-annex-shell.)
"""]]


code:
What steps will reproduce the problem?

Attempting to pair between a local repository and a repository on a remote computer on my LAN. Pairing is initiated from my local machine and I'm interacting with the webapp on the remote machine via firefox running over an ssh -X connection. Pairing appears to work up to a point: I enter the secret at one end, the pairing request shows up at the other end. I then enter the secret at that end.

What is the expected output? What do you see instead?

Pairing should complete successfully. Instead I get the error message "PairListener crashed: bad comment in public key", followed by the public key. The pairing process then does not move beyond the 'awaiting pairing' pages.

What version of git-annex are you using? On what operating system?

Local Machine: 3.20121127, Debian Wheezy/Sid (the only package from unstable is git-annex).
Remote Machine: 3.20121113, Arch Linux (I installed the version from: https://aur.archlinux.org/packages/git-annex-bin/, which is supposedly the same as above, but reports the version specified here).

Please provide any additional information below.

None as yet. Let me know if there are any log files, etc. that I can post.

> So it was the period in the hostname! [[fixed|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.72"
 subject="comment 1"
 date="2012-12-04T17:38:47Z"
 content="""
On the machine that didn't crash, run:

ssh-keygen -P \"\" -f test; cat test.pub

Probably the non-alphanumeric character is part of the machine's hostname, or perhaps your username. We'll see..
"""]]


code:
[[!comment format=mdwn
 username="robconnolly"
 ip="118.90.115.19"
 subject="comment 2"
 date="2012-12-06T02:29:56Z"
 content="""
Hi Joey,

Thanks for your response. Here is the output:

[robert@laforge ~]$ ssh-keygen -P \"\" -f test; cat test.pub
Generating public/private rsa key pair.
Your identification has been saved in test.
Your public key has been saved in test.pub.
The key fingerprint is:
84:43:aa:f0:ba:a7:f0:11:41:55:3b:c4:63:6a:71:e1 robert@laforge.enterprise
The key's randomart image is:
+--[ RSA 2048]----+
|  ...o=.         |
| .  .==o         |
|. . .=E..        |
| o oo  +         |
|  +.    S        |
| . .             |
|o .              |
|.o..             |
|oo.              |
+-----------------+
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIeitbarby1thLPEyVpeT/vDaWt0MJwLPvhT7TEZv0FedYalvz/mm3enULHiYdxkjWGBaO/EEzfmz5bjhoJx2tsqSLsUj/UZVv0LxsSfZwTrk0SPuUerzWvAJAXKvtPobDXJoAxsNvzSJN392BMOLKqcjAQTqPA3vd5+EjUVIgJxguz5poGrP0ZRMQD5LmsDrPGUWNJ/EHaVkrqGobAE4DVr8vhJTY41h5Gk2ipnzx+GD6CDZTSNFwl8RorjFAV3mYTTjHc2Cz7GxTLSmwF0+qFZ/UkiO2tD1M37Y4/cPLY7GpVfKvpPHTC8pJUpvj+B4uujHxvZkBJLLz9XhH9KsZ robert@laforge.enterprise

Let me know if you need any more info.
"""]]


code:
### Please describe the problem.

Creating a USB repo fails with a GPG error.

### What steps will reproduce the problem?

  * Build git-annex and git-annex assistant using the instructions at https://gist.github.com/calmyournerves/7144127
  * Run git-annex app to launch web interface
  * Create local repo using web interface
  * Try to create USB repo using web interface

### What version of git-annex are you using? On what operating system?

git-annex version 4.20131105-g136b030 on Mac OS 10.9 Mavericks.

### Please provide any additional information below.

[[!format sh """
07/Nov/2013:06:51:07 +1100 [Error#yesod-core] user error (gpg ["--batch","--no-tty","--use-agent","--quiet","--trust-model","always","--with-colons","--list-secret-keys","--fixed-list-mode"] exited 5) @(yesod-core-1.2.4.5:Yesod.Core.Class.Yesod ./Yesod/Core/Class/Yesod.hs:485:5)
"""]]

> [[fixed|done]]; it seems that this was a local build issue ad does not
> affect the autobuild. --[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmZgZuUhZlHpd_AbbcixY0QQiutb2I7GWY"
 nickname="Jimmy"
 subject="Updated version 4.20131105-g136b030 seems to work"
 date="2013-11-07T20:28:55Z"
 content="""
I've updated to the latest version as of 12 hours ago and I was able to create a USB repo on two computers.  4.20131105-g136b030 showing on the web app but 20131106 on the command line.

Is there an easy way of checking the repo is correctly encrypted?  I didn't see anything about encrypting (now the previous error has disappeared) when creating the repo - is this what I should expect?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 2"
 date="2013-11-08T17:36:53Z"
 content="""
USB repos are not set up encryption unless you explicitly request it.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmZgZuUhZlHpd_AbbcixY0QQiutb2I7GWY"
 nickname="Jimmy"
 subject="Yes, I just figured that out"
 date="2013-11-08T22:09:40Z"
 content="""
I figured it out a couple of hours ago when I was able to use the existing USB repo on a new computer that doesn't have any of my private keys on it.

I didn't see an option in the web interface to use encryption - is it command line only?
"""]]


code:
### Please describe the problem.

Creating a remote S3 repository using the git-annex assistant web interface fails with a GPG error.  (I'm also getting a GPG error trying to create a USB repo but it's slightly different so I'll post a different bug.)

### What steps will reproduce the problem?

  * Build git-annex and git-annex assistant using the instructions at https://gist.github.com/calmyournerves/7144127
  * Run git-annex app to launch web interface
  * Create local repo using web interface
  * Try to create encrypted S3 remote repo using web interface

### What version of git-annex are you using? On what operating system?

git-annex version 4.20131105-g136b030 on MacOS 10.9 Mavericks.

### Please provide any additional information below.

[[!format sh """

(encryption setup) dyld: Library not loaded: @rpath/libz.1.2.8.dylib
  Referenced from: /Applications/git-annex.app/Contents/MacOS/bundle/gpg
  Reason: image not found
07/Nov/2013:06:38:27 +1100 [Error#yesod-core] user error (gpg ["--batch","--no-tty","--use-agent","--quiet","--trust-model","always","--gen-random","--armor","1","512"] exited 5) @(yesod-core-1.2.4.5:Yesod.Core.Class.Yesod ./Yesod/Core/Class/Yesod.hs:485:5)

"""]]

> [[fixed|done]]; it seems that this was a local build issue ad does not
> affect the autobuild. --[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmZgZuUhZlHpd_AbbcixY0QQiutb2I7GWY"
 nickname="Jimmy"
 subject="S3 works without encryption"
 date="2013-11-06T21:09:26Z"
 content="""
Not surprisingly, S3 repos work without encryption.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmZgZuUhZlHpd_AbbcixY0QQiutb2I7GWY"
 nickname="Jimmy"
 subject="Still fails with git-annex version 4.20131105-g136b030"
 date="2013-11-07T20:32:09Z"
 content="""
Updating seems to have fixed my other problem with creating a USB repo but still fails when trying to create an encrypted S3 repo.

    (encryption setup) dyld: Library not loaded: @rpath/libz.1.2.8.dylib
      Referenced from: /Applications/git-annex.app/Contents/MacOS/bundle/gpg
      Reason: image not found
    08/Nov/2013:07:30:11 +1100 [Error#yesod-core] user error (gpg [\"--batch\",\"--no-tty\",\"--use-agent\",\"--quiet\",\"--trust-model\",\"always\",\"--gen-random\",\"--armor\",\"1\",\"512\"] exited 5) @(yesod-core-1.2.4.5:Yesod.Core.Class.Yesod ./Yesod/Core/Class/Yesod.hs:485:5)
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 3"
 date="2013-11-08T17:38:40Z"
 content="""
Looks like it failed to include libz.1.2.8.dylib in the bundle for some reason despite gpg needing it.

I don't really see the point in building an app bundle if you're going to install it back to the same machine you built it on. It's much easier to just `cabal install git-annex` in this case.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmZgZuUhZlHpd_AbbcixY0QQiutb2I7GWY"
 nickname="Jimmy"
 subject="Maybe I'm missing something..."
 date="2013-11-08T22:12:04Z"
 content="""
But does cabal install git-annex install the assistant?  On the Mac, I use the app bundle to launch the web interface from my applications folder.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmZgZuUhZlHpd_AbbcixY0QQiutb2I7GWY"
 nickname="Jimmy"
 subject="To answer my own question - "git annex webapp""
 date="2013-11-09T00:10:28Z"
 content="""
Will open the web app from the command line.

But now I get a different error trying to create an encrypted S3 repo.

    (encryption setup) 09/Nov/2013:11:08:56 +1100 [Error#yesod-core] user error (gpg [\"--quiet\",\"--trust-model\",\"always\",\"--gen-random\",\"--armor\",\"1\",\"512\"] exited 127) @(yesod-core-1.2.4.5:Yesod.Core.Class.Yesod ./Yesod/Core/Class/Yesod.hs:485:5)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmZgZuUhZlHpd_AbbcixY0QQiutb2I7GWY"
 nickname="Jimmy"
 subject="Works with git annex on command line"
 date="2013-11-09T20:18:38Z"
 content="""
git annex initremote cloud type=S3 keyid=key correct creates an encrypted S3 repo and I'm able to upload files.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.246"
 subject="comment 7"
 date="2013-11-15T19:00:25Z"
 content="""
It would be useful if you could test with the 10.9 build now available in [[install/OSX]].
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmZgZuUhZlHpd_AbbcixY0QQiutb2I7GWY"
 nickname="Jimmy"
 subject="comment 8"
 date="2013-11-15T21:31:40Z"
 content="""
Thanks Joey - I was able to create an encrypted S3 repo using the new Mavericks build.  Files are transferring across and are encrypted.
"""]]


code:
Latest build for Mac OS X (both autobuild and release versions) does not contain webapp.

git annex version for OS X,

    git-annex version: 5.20140306-g309a73c
    build flags: Assistant Pairing Testsuite S3 WebDAV FsEvents XMPP DNS Feeds Quvi TDFA CryptoHash
    key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN2 56 SKEIN512 WORM URL
    remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier hook external

whereas on my Linux box build flags include webapp. On os x when I run git annex webapp it does nothing, just prints the help info.

> [[fixed|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.146"
 subject="comment 1"
 date="2014-03-07T15:17:16Z"
 content="""
I've fixed the missing warp-tls dep on the autobuilder and updated the builds.
"""]]


code:
### Please describe the problem.

Joey, it looks like the git version wasn't updated with the latest release as is still too old to respect .gitignore files.  I'm hoping that I haven't just made a silly mistake but I don't think I have...

See http://git-annex.branchable.com/bugs/Mac_OS_git_version_too_old_to_honour_.gitignore/ for bug that was closed.

### What steps will reproduce the problem?

Install git-annex 5.20140209-g3a61dbe and try to use .gitignore file to exclude items from git annex.

### What version of git-annex are you using? On what operating system?

5.20140209-g3a61dbe on Mac OS 10.9.1.

### Please provide any additional information below.

[/Applications/git-annex.app/Contents/MacOS]# ./git annex version

git-annex version: 5.20140209-g3a61dbe
build flags: Assistant Webapp Pairing S3 WebDAV FsEvents XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier hook external

[/Applications/git-annex.app/Contents/MacOS]# ./git --version

git version 1.8.3.4 (Apple Git-47)

> Really [[fixed|done]] now. --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.172"
 subject="comment 1"
 date="2014-02-20T18:59:44Z"
 content="""
I thought it got upgrades, perhaps it was downgraded again? I have prodded Kevin.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmZgZuUhZlHpd_AbbcixY0QQiutb2I7GWY"
 nickname="Jimmy"
 subject="comment 2"
 date="2014-02-21T07:03:20Z"
 content="""
Thanks Joey and Kevin.  Glad to have the bug really fixed and glad to know that I wasn't missing something obvious!
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmZgZuUhZlHpd_AbbcixY0QQiutb2I7GWY"
 nickname="Jimmy"
 subject="comment 3"
 date="2014-02-21T22:05:06Z"
 content="""
And yep, it's fixed in 5.20140221-g1a47f5f.  Thanks guys!
"""]]


code:
### Please describe the problem.

Git annex assistant ignores .gitignore file due to packaged git version being too old.

I have a locally installed version that is greater than the 1.8.4 needed to respect .gitignore but git annex doesn't use it.

### What steps will reproduce the problem?

- Create local repository using webapp
- Add .gitignore file to local repository
- Add files that match .gitignore patterns and watch git annex add them

### What version of git-annex are you using? On what operating system?

Git Annex assistant version 5.20140128-g0ac94c3 on Mac OS 10.9.1

### Please provide any additional information below.

Log message is "The installed version of git is too old for .gitignores to be honored by git-annex."

[[!format sh """
# /Applications/git-annex.app/Contents/MacOS/git-annex version
git-annex version: 5.20140128-g0ac94c3
build flags: Assistant Webapp Pairing S3 WebDAV FsEvents XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier hook external

# /Applications/git-annex.app/Contents/MacOS/git --version
git version 1.8.3.4 (Apple Git-47)

# which git
/usr/local/bin/git

# /usr/local/bin/git --version
git version 1.8.5.3
"""]]

> [[fixed|done]]; it has been updated to 1.8.5.3 on the autobuilder. --[[Joey]]


code:
### Please describe the problem.

The `manual` content mode doesn't follow the description provided in the help page, instead it seems to collect content.

### What steps will reproduce the problem?

1. Create a new git annex repository using the webapp, set the content type to `client`.
2. Create another repository, and set the content type to `manual`.
3. Copy something into the `client` repository.
4. It will be pushed/pulled into the `manual` repository.

### What version of git-annex are you using? On what operating system?

    git-annex version: 4.20130521-g25dba9d
    Ubuntu 13.04 x64.

### Please provide any additional information below.

I have also noticed very weird behaviour that I have been unable to replicate in testing, but I will describe the setup that it currently happens in:
I have 3x repositories, one a `client` repository, and the other two are set to `manual`. When put a new file into the `client` repository, it is pushed onto the two `manual` repositories. When these repositories have received it, the client drops the file and re-downloads it from one of the `manual` repositories. Once it's been pushed, deleted, and pulled, everything is happy... but the extra step makes no difference.

[[!format sh """
[2013-05-22 20:41:44 EST] main: starting assistant version 4.20130521-g25dba9d
[2013-05-22 20:41:44 EST] TransferScanner: Syncing with test3, test2
Already up-to-date.

(scanning...) [2013-05-22 20:41:44 EST] Watcher: Performing startup scan
Already up-to-date.
Already up-to-date.

(started...) From /home/valorin/workspace/tmp/test3
   f285dc2..406c20c  git-annex  -> test3/git-annex
   cdf2ad3..508983c  master     -> test3/master
From /home/valorin/workspace/tmp/test2
   1e04829..1c03533  git-annex  -> test2/git-annex
   8ad4bd3..18a5408  master     -> test2/master
Updating 508983c..18a5408
Fast-forward
Already up-to-date.
To /home/valorin/workspace/tmp/test2
   4e49293..a66ce5d  git-annex -> synced/git-annex
   508983c..18a5408  master -> synced/master
To /home/valorin/workspace/tmp/test3
   4e49293..a66ce5d  git-annex -> synced/git-annex
   508983c..18a5408  master -> synced/master
Already up-to-date.
Already up-to-date.
[2013-05-22 20:42:07 EST] Committer: Adding Firefly S..acked.m4v

(merging test3/git-annex into git-annex...)
(merging test2/git-annex into git-annex...)
(Recording state in git...)

add Firefly S01E03 Bushwhacked.m4v (checksum...) [2013-05-22 20:42:15 EST] Committer: Committing changes to git
[2013-05-22 20:42:15 EST] Pusher: Syncing with test3, test2
Already up-to-date.
To /home/valorin/workspace/tmp/test2
   a66ce5d..a6773dd  git-annex -> synced/git-annex
   18a5408..f9e7692  master -> synced/master
To /home/valorin/workspace/tmp/test3
   a66ce5d..a6773dd  git-annex -> synced/git-annex
   18a5408..f9e7692  master -> synced/master
Already up-to-date.
Already up-to-date.
[2013-05-22 20:42:26 EST] Transferrer: Uploaded Firefly S..acked.m4v
[2013-05-22 20:42:26 EST] Pusher: Syncing with test3, test2
To /home/valorin/workspace/tmp/test3
   a6773dd..c35f992  git-annex -> synced/git-annex
To /home/valorin/workspace/tmp/test2
   a6773dd..c35f992  git-annex -> synced/git-annex
[2013-05-22 20:42:35 EST] Transferrer: Uploaded Firefly S..acked.m4v
[2013-05-22 20:42:35 EST] Pusher: Syncing with test3, test2
To /home/valorin/workspace/tmp/test3
   c35f992..9e47813  git-annex -> synced/git-annex
To /home/valorin/workspace/tmp/test2
   c35f992..9e47813  git-annex -> synced/git-annex
[2013-05-22 20:42:44 EST] Pusher: Syncing with test3, test2
Everything up-to-date
Everything up-to-date
"""]]

> It turns out there was a bug in the preferred content expression parser,
> that made it parse the expression for manual mode (but I think no other standard
> expression) quite wrong, as if it had parens in the wrong place. This explains
> the broken behavior. [[fixed|done]] --[[Joey]]


code:
### Please describe the problem.
When creating a symlink in repository A, and creating a regular file under the same name in repository B, syncing B will yield the result that the symlink is lost, and both the original filename and the .variant file will point to the same annex object containing the original content from B.

Both A and B are indirect mode repos.

### What steps will reproduce the problem?

[[!format  sh """

#Initial state:

repo-A$ echo file1
This is file 1.
repo-B$ echo file1
This is file 1.

#Make conflicting changes:

repo-A$ ln -s file1 file2; git add file2; git commit -m "Add file2 as symlink."
repo-B$ echo "This is file 2." > file2; git annex add file2; git commit -m "Add file2 as regular file."

#Sync it:

repo-A$ git annex sync
repo-B$ git annex sync

#Strange result in repo-B:

repo-B$ ls -l file2*
file2 -> .git/annex/objects/$HASH1
file2.variant1234 -> .git/annex/objects/$HASH1
repo-B$ cat file2 file2.variantXXXX
This is file 2.
This is file 2.

#Repo-A leaves the symlink change untouched and adds a .variant containing the new regular file data.

repo-A$ ls -l file*
file2 -> file1
file2.variant1234 -> .git/annex/objects/$HASH1
repo-A$ cat file.variant1234
This is file 2.
"""]]
### What version of git-annex are you using? On what operating system?
Linux 3.15.3
git-annex  5.20140613

[[!tag confirmed]]

> [[fixed|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="zardoz"
 ip="134.147.14.84"
 subject="comment 1"
 date="2014-07-07T14:09:33Z"
 content="""
Sorry, the initial «echos» should have been «cat» of course.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.55"
 subject="comment 2"
 date="2014-07-07T17:17:49Z"
 content="""
Drat, so many bug fixes and test cases and this still got through?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.55"
 subject="comment 3"
 date="2014-07-07T17:19:03Z"
 content="""
Ah, I see, it's explicitly because the non-git-annex symlink is involved. Whew!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.55"
 subject="analysis"
 date="2014-07-08T17:52:59Z"
 content="""
When resolveMerge' calls graftin to add A's file2 symlink to B's tree, it actually stages the right symlink (the non-annexed one).

However, the work tree is left as-is, so it still has the annexed symlink in it. So git status shows file2 as modified. Later syncs will commit that.

This is why the sync in A doesn't have the problem, as there things are the other way around, and git-annex makes the git-annex symlink, leaving the non-annexed symlink as-is in the work tree.

So, graftin needs to update the work tree. But it's tricky because graftin is called in 3 situations: non-symlink file, directory, and non-annexed symlink.
Interestingly, in the other 2 cases, git-merge already takes care of updating the work tree -- it deletes the annexed symlink and puts in place either the non-symlink file or the directory. It's only the the case of a merge conflict involving 2 symlinks that git merge doesn't update the tree in this way. It's nice to be able to rely on git-merge in the other 2 cases, especially the directory case (avoids having to manually check out the directory).
"""]]


code:
[[!comment format=mdwn
 username="zardoz"
 ip="78.49.247.112"
 subject="comment 5"
 date="2014-07-08T19:17:29Z"
 content="""
Thanks for the swift fix + analysis! If I’m ever around, I’ll lend you a hand on your next truckload of firewood! ;>

Cheers!
"""]]


code:
### Please describe the problem.
I paired my repo running on Gentoo (git-annex 4.20130601) with Ubuntu 13.04 (git-annex 3.10121112ubuntu4). The repo on Ubuntu doesn't have uuid for the Gentoo remote, so:

- There is no name in assistan's repo settings for it

- Trying to access its settings gives Internal server error: Unknown UUID
15/Jun/2013:12:39:10 +0300 [Error#yesod-core] Unknown UUID @(yesod-core-1.1.8.3:Yesod.Internal.Core ./Yesod/Internal/Core.hs:550:5)

- In dashboard on Ubuntu all changes stay queued forever (although the syncing seems to work)

### What steps will reproduce the problem?
Pair local computers with different annex versions.

### What version of git-annex are you using? On what operating system?
Gentoo (git-annex 4.20130601)
Ubuntu 13.04 (git-annex 3.10121112ubuntu4)

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

# End of transcript or log.
"""]]

> Others are reporting what seems to be the same problem here:
> [[Internal_Server_Error:_Unknown_UUID]].
>
> [[dup|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.193"
 subject="comment 1"
 date="2013-06-25T18:08:55Z"
 content="""
As far as I know, there have been no changes to the local pairing protocol that would cause a problem like this. 3.20121112 is a few months after local pairing was first developed.

Have you tried upgrading and re-pairing?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm01ida6POv7vqyUYtOlymEbJTbrImAIzM"
 nickname="Reinis"
 subject="comment 2"
 date="2013-06-25T18:50:00Z"
 content="""
I have not tried to upgrade, but I fixed it using annex describe followed by assistant restart as hinted in /usr/share/doc/git-annex/html/bugs/uuid.log_trust.log_and_remote.log_merge_wackiness.html.

From that document I understood that at worst it should use one of the two uuids (from Ubuntu or Gentoo repo), but it should not leave it without uuid and hence this report.

I have tried it a couple of times and it is perfectly reproducible.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.193"
 subject="comment 3"
 date="2013-06-25T19:13:06Z"
 content="""
An old bug report that was fixed in 2011 seems very unlikely to have anything to do with your local pairing problem.

Why don't you try upgrading?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm01ida6POv7vqyUYtOlymEbJTbrImAIzM"
 nickname="Reinis"
 subject="comment 4"
 date="2013-06-25T19:34:12Z"
 content="""
I was expecting the version in distribution which came out two months ago to be somewhat recent, probably unfounded expectation for non-rolling release distribution. I'll add PPA and try using that one.
"""]]


code:
### Please describe the problem.
I have managed to get most things working under OpenBSD 5.4 now.

One of the last hurdles is that if I enable XMPP the build fails on "Loading package gnuidn-0.2.1..."
See the error below.

I suspect this is an error in git-annex because network-protocol-xmpp AND gnuidn compiles (and links) fine.

I will gladly do anything I can to get this working, but I'm at a loss what to do right now. It's the last major piece of the puzzle before I get it properly functioning under OpenBSD.

### What steps will reproduce the problem?
Building with XMPP support under OpenBSD 5.4

### What version of git-annex are you using? On what operating system?
5.20140129 under OpenBSD 5.4

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
Loading package gnuidn-0.2.1 ...

GHCi runtime linker: fatal error: I found a duplicate definition for symbol
   c_isascii
whilst processing object file
   /usr/local/lib/libidn.a
This could be caused by:
   * Loading two different object files which export the same symbol
   * Specifying the same object file twice on the GHCi command line
   * An incorrect `package.conf' entry, causing some object to be
     loaded twice.
GHCi cannot safely continue in this situation.  Exiting now.  Sorry.

# End of transcript or log.
"""]]

[[!tag moreinfo]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.163"
 subject="comment 10"
 date="2014-02-08T18:31:23Z"
 content="""
But you said that setSocketOption failed when you were using XMPP, not when starting the webapp, so I think it's more likely to be one of the setSocketOption calls in the network library, or possibly somewhere else.
"""]]