code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 2"
 date="2014-03-26T22:39:34Z"
 content="""
Sorry, I had to remove the fixed tag.

The bug makes sense, as does your fix. I didn't even consider that this may be the cause. Still, `git remote rm` and `git annex forget --drop-dead --force` does not seem to be enough to truly get rid of the repo (and its UUID & state):

[[!format sh \"\"\"
richih@apu [2] [0] (git)-[master] /srv/video/video.debian.net % git annex info
repository mode: indirect
trusted repositories: 0
semitrusted repositories: 4
        00000000-0000-0000-0000-000000000001 -- web
        070cff8a-6302-4aa7-a63c-3fdd34e598a2 -- amazon_s3_us_east--SHA512E
        0bae683f-bede-43dd-a815-c4f8fb6db32d -- aws_s3_us_east--SHA512E
        92e9fac9-97ec-401f-a421-33f6b4f43e47 -- here (apu.debconf.org/srv/video/conference_videos.annex)
untrusted repositories: 0
transfers in progress: none
available local disk space: 136.3 gigabytes (+1 megabyte reserved)
local annex keys: git4392
local annex size: 884.64 gigabytes
annexed files in working tree:  am4628
size of annexed files in working tree: 885.68 gigabytes
bloom filter size: 16 mebibytes (0.9% full)
backend usage:
        SHA512E: 9020
richih@apu [2] [0] (git)-[master] /srv/video/video.debian.net % git annex dead amazon_s3_us_east--SHA512E
dead amazon_s3_us_east--SHA512E ok
(Recording state in git...)
richih@apu [2] [0] (git)-[master] /srv/video/video.debian.net % git annex dead aws_s3_us_east--SHA512E
dead aws_s3_us_east--SHA512E ok
(Recording state in git...)
richih@apu [2] [0] (git)-[master] /srv/video/video.debian.net % git remote rm amazon_s3_us_east--SHA512E
richih@apu [2] [0] (git)-[master] /srv/video/video.debian.net % git remote rm aws_s3_us_east--SHA512E
richih@apu [2] [0] (git)-[master] /srv/video/video.debian.net % git annex forget --drop-dead --force
forget git-annex (Recording state in git...)
ok
(Recording state in git...)
richih@apu [2] [0] (git)-[master] /srv/video/video.debian.net % git annex info
repository mode: indirect
trusted repositories: 0
semitrusted repositories: 2
        00000000-0000-0000-0000-000000000001 -- web
        92e9fac9-97ec-401f-a421-33f6b4f43e47 -- here (apu.debconf.org/srv/video/conference_videos.annex)
untrusted repositories: 0
transfers in progress: none
available local disk space: 136.3 gigabytes (+1 megabyte reserved)
local annex keys: 4392
local annex size: 884.64 gigabytes
annexed files in working tree: 4628
size of annexed files in working tree: 885.68 gigabytes
bloom filter size: 16 mebibytes (0.9% full)
backend usage:
        SHA512E: 9020
richih@apu [2] [0] (git)-[master] /srv/video/video.debian.net % git annex initremote amazon_aws_s3-us_east_1--SHA512E type=S3 encryption='none' embedcreds='no' fileprefix='SHA512E/' bucket='debian-video'
initremote amazon_aws_s3-us_east_1--SHA512E (checking bucket...) git-annex: This bucket is already in use by a different S3 special remote, with UUID: 0bae683f-bede-43dd-a815-c4f8fb6db32d
richih@apu [2] [0] (git)-[master] /srv/video/video.debian.net % cat .git/annex
cat: .git/annex: Ist ein Verzeichnis
richih@apu [2] [0] (git)-[master] /srv/video/video.debian.net % cat .git/config
[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
[annex]
        uuid = 92e9fac9-97ec-401f-a421-33f6b4f43e47
        version = 5
richih@apu [2] [0] (git)-[master] /srv/video/video.debian.net % git annex info                                                                                537 23:33:37 Mi 26.03.2014
repository mode: indirect
trusted repositories: 0
semitrusted repositories: 2
        00000000-0000-0000-0000-000000000001 -- web
        92e9fac9-97ec-401f-a421-33f6b4f43e47 -- here (apu.debconf.org/srv/video/conference_videos.annex)
untrusted repositories: 0
transfers in progress: none
available local disk space: 136.3 gigabytes (+1 megabyte reserved)
local annex keys: 4392
local annex size: 884.64 gigabytes
annexed files in working tree: 4628
size of annexed files in working tree: 885.68 gigabytes
bloom filter size: 16 mebibytes (0.9% full)
backend usage:
        SHA512E: 9020
richih@apu [2] [0] (git)-[master] /srv/video/video.debian.net %
\"\"\"]]
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.41"
 subject="comment 3"
 date="2014-03-27T17:40:51Z"
 content="""
git-annex stores the uuid of a S3 remote inside the bucket for various reasons. Now that you have removed all knowledge of the remote from the repository, when you attempt to reuse the same bucket for some reason, git-annex has no way to know that this is a remote it used to use with that bucket.

I think this behavior is entirely reasonable. Also, it's not what you filed the original bug report about; I fixed that bug. I'm going to re-close this.
"""]]

repo_name: kakafarm/git

license: unknown


code:
### Please describe the problem.

mtimes are clobbered with what I think is the time of the first time `git annex` saw a file in that directory on all files which are `git annex import`ed.

### What steps will reproduce the problem?

    richih@eudyptes (git)-[master] ~/killme/target % date; stat ../source/*; date; git annex import ../source/*; date; stat *; date
    Tue Jul 30 01:41:43 CEST 2013
      File: ‘../source/foo’
      Size: 0           Blocks: 0          IO Block: 4096   regular empty file
    Device: 804h/2052d  Inode: 110095      Links: 1
    Access: (0644/-rw-r--r--)  Uid: ( 1000/  richih)   Gid: ( 1000/  richih)
    Access: 2013-07-30 01:41:03.860703428 +0200
    Modify: 2013-07-30 01:41:03.860703428 +0200
    Change: 2013-07-30 01:41:03.860703428 +0200
     Birth: -
      File: ‘../source/foobar’
      Size: 0           Blocks: 0          IO Block: 4096   regular empty file
    Device: 804h/2052d  Inode: 110104      Links: 1
    Access: (0644/-rw-r--r--)  Uid: ( 1000/  richih)   Gid: ( 1000/  richih)
    Access: 2013-07-30 01:41:05.512703349 +0200
    Modify: 2013-07-30 01:41:05.512703349 +0200
    Change: 2013-07-30 01:41:05.512703349 +0200
     Birth: -
    Tue Jul 30 01:41:43 CEST 2013
    import foo (checksum...) ok
    import foobar (checksum...) ok
    (Recording state in git...)
    Tue Jul 30 01:41:43 CEST 2013
      File: ‘foo’ -> ‘.git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855’
      Size: 178         Blocks: 8          IO Block: 4096   symbolic link
    Device: 804h/2052d  Inode: 268638577   Links: 1
    Access: (0777/lrwxrwxrwx)  Uid: ( 1000/  richih)   Gid: ( 1000/  richih)
    Access: 2013-07-30 01:41:43.704701516 +0200
    Modify: 2013-07-30 01:19:41.000000000 +0200
    Change: 2013-07-30 01:41:43.700701516 +0200
     Birth: -
      File: ‘foobar’ -> ‘.git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855’
      Size: 178         Blocks: 8          IO Block: 4096   symbolic link
    Device: 804h/2052d  Inode: 268774374   Links: 1
    Access: (0777/lrwxrwxrwx)  Uid: ( 1000/  richih)   Gid: ( 1000/  richih)
    Access: 2013-07-30 01:41:43.704701516 +0200
    Modify: 2013-07-30 01:19:41.000000000 +0200
    Change: 2013-07-30 01:41:43.700701516 +0200
     Birth: -
    Tue Jul 30 01:41:43 CEST 2013
    richih@eudyptes (git)-[master] ~/killme/target %

### What version of git-annex are you using? On what operating system?

    git-annex version: 4.20130709
    build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP
    local repository version: 3
    default repository version: 3
    supported repository versions: 3 4
    upgrade supported from repository versions: 0 1 2

Debian unstable amd64

> [[fixed|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 1"
 date="2013-07-29T23:49:12Z"
 content="""
PS: Maybe adding tests that _all_ commands leave mtime in place would be an option? As it is, I keep (for some value of) finding places where mtimes are clobbered.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.21"
 subject="comment 2"
 date="2013-07-30T17:40:04Z"
 content="""
git-annex makes only a best-effort attempt to preserve mtime. There are multiple cases where it cannot:

* filesystems that do not support changing the mtime of a symlink
* special remotes
* when 2 files with the same content get stored in the annex, the mtime of one must win over the other since we only have 1 inode

This last seems to be what has happened in the example you show. Both `foo` and `foobar` are empty. I'm not sure where the 01:19:41 time came from -- perhaps you had a third empty file that was not shown?

I've tested `git annex import` and it preserves mtimes otherwise.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 3"
 date="2013-08-02T06:57:12Z"
 content="""
In this particular case

* It's XFS
* It's a normal repo
* That the inode can have only one shared mtime is obvious, but the symlinks could have their own:

Random line to make MarkDown happy

    richih@eudyptes (git)-[master] ~/killme/target % stat * .git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
      File: ‘foo’ -> ‘.git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855’
      Size: 178          Blocks: 8          IO Block: 4096   symbolic link
    Device: 804h/2052d   Inode: 268638577   Links: 1
    Access: (0777/lrwxrwxrwx)  Uid: ( 1000/  richih)   Gid: ( 1000/  richih)
    Access: 2013-08-02 08:51:35.976674954 +0200
    Modify: 2013-07-30 01:19:41.000000000 +0200
    Change: 2013-07-30 01:41:43.700701516 +0200
     Birth: -
      File: ‘foobar’ -> ‘.git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855’
      Size: 178          Blocks: 8          IO Block: 4096   symbolic link
    Device: 804h/2052d   Inode: 268774374   Links: 1
    Access: (0777/lrwxrwxrwx)  Uid: ( 1000/  richih)   Gid: ( 1000/  richih)
    Access: 2013-08-02 08:51:35.976674954 +0200
    Modify: 2013-07-30 01:19:41.000000000 +0200
    Change: 2013-07-30 01:41:43.700701516 +0200
     Birth: -
      File: ‘.git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855’
      Size: 0            Blocks: 0          IO Block: 4096   regular empty file
    Device: 804h/2052d   Inode: 73750       Links: 1
    Access: (0444/-r--r--r--)  Uid: ( 1000/  richih)   Gid: ( 1000/  richih)
    Access: 2013-07-30 01:20:11.915507749 +0200
    Modify: 2013-07-30 01:19:41.131509226 +0200
    Change: 2013-07-30 01:20:11.915507749 +0200
     Birth: -
    richih@eudyptes (git)-[master] ~/killme/target % touch foo
    richih@eudyptes (git)-[master] ~/killme/target % stat * .git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
      File: ‘foo’ -> ‘.git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855’
      Size: 178          Blocks: 8          IO Block: 4096   symbolic link
    Device: 804h/2052d   Inode: 268638577   Links: 1
    Access: (0777/lrwxrwxrwx)  Uid: ( 1000/  richih)   Gid: ( 1000/  richih)
    Access: 2013-08-02 08:51:35.976674954 +0200
    Modify: 2013-07-30 01:19:41.000000000 +0200
    Change: 2013-07-30 01:41:43.700701516 +0200
     Birth: -
      File: ‘foobar’ -> ‘.git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855’
      Size: 178          Blocks: 8          IO Block: 4096   symbolic link
    Device: 804h/2052d   Inode: 268774374   Links: 1
    Access: (0777/lrwxrwxrwx)  Uid: ( 1000/  richih)   Gid: ( 1000/  richih)
    Access: 2013-08-02 08:51:35.976674954 +0200
    Modify: 2013-07-30 01:19:41.000000000 +0200
    Change: 2013-07-30 01:41:43.700701516 +0200
     Birth: -
      File: ‘.git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855’
      Size: 0            Blocks: 0          IO Block: 4096   regular empty file
    Device: 804h/2052d   Inode: 73750       Links: 1
    Access: (0444/-r--r--r--)  Uid: ( 1000/  richih)   Gid: ( 1000/  richih)
    Access: 2013-08-02 08:54:34.464668766 +0200
    Modify: 2013-08-02 08:54:34.464668766 +0200
    Change: 2013-08-02 08:54:34.464668766 +0200
     Birth: -
    richih@eudyptes (git)-[master] ~/killme/target % touch --no-dereference foo
    richih@eudyptes (git)-[master] ~/killme/target % stat * .git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
      File: ‘foo’ -> ‘.git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855’
      Size: 178          Blocks: 8          IO Block: 4096   symbolic link
    Device: 804h/2052d   Inode: 268638577   Links: 1
    Access: (0777/lrwxrwxrwx)  Uid: ( 1000/  richih)   Gid: ( 1000/  richih)
    Access: 2013-08-02 08:55:07.796667610 +0200
    Modify: 2013-08-02 08:55:07.796667610 +0200
    Change: 2013-08-02 08:55:07.796667610 +0200
     Birth: -
      File: ‘foobar’ -> ‘.git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855’
      Size: 178          Blocks: 8          IO Block: 4096   symbolic link
    Device: 804h/2052d   Inode: 268774374   Links: 1
    Access: (0777/lrwxrwxrwx)  Uid: ( 1000/  richih)   Gid: ( 1000/  richih)
    Access: 2013-08-02 08:51:35.976674954 +0200
    Modify: 2013-07-30 01:19:41.000000000 +0200
    Change: 2013-07-30 01:41:43.700701516 +0200
     Birth: -
      File: ‘.git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855’
      Size: 0            Blocks: 0          IO Block: 4096   regular empty file
    Device: 804h/2052d   Inode: 73750       Links: 1
    Access: (0444/-r--r--r--)  Uid: ( 1000/  richih)   Gid: ( 1000/  richih)
    Access: 2013-08-02 08:54:34.464668766 +0200
    Modify: 2013-08-02 08:54:34.464668766 +0200
    Change: 2013-08-02 08:54:34.464668766 +0200
     Birth: -
    richih@eudyptes (git)-[master] ~/killme/target %

As to where the initial time of 01:19:41 came from: Yes, those were earlier tests with other imports of empty files.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 4"
 date="2013-09-26T09:08:11Z"
 content="""
Fixed in http://source.git-annex.branchable.com/?p=source.git;a=commitdiff;h=98fc7e8
"""]]


code:
### Please describe the problem.

Building a recent git-annex with cabal with the `minimal build'
options given in the installation instructions fails. It is probably
just a matter of fixing the dependencies in the cabal file.

### What steps will reproduce the problem?
Compile with:

cabal install git-annex-5.20140108 --bindir=$HOME/bin -f"-assistant -webapp -webdav -pairing -xmpp -dns"

### What version of git-annex are you using? On what operating system?
Linux 2.6.32-5-686 i686 GNU/Linux

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
Resolving dependencies...
[ 1 of 27] Compiling Utility.PartialPrelude ( /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/Utility/PartialPrelude.hs, /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/dist/setup/Utility/PartialPrelude.o )
[ 2 of 27] Compiling Utility.FileSystemEncoding ( /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/Utility/FileSystemEncoding.hs, /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/dist/setup/Utility/FileSystemEncoding.o )
[ 3 of 27] Compiling Utility.Applicative ( /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/Utility/Applicative.hs, /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/dist/setup/Utility/Applicative.o )
[ 4 of 27] Compiling Utility.Data     ( /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/Utility/Data.hs, /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/dist/setup/Utility/Data.o )
[ 5 of 27] Compiling Utility.Exception ( /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/Utility/Exception.hs, /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/dist/setup/Utility/Exception.o )
[ 6 of 27] Compiling Utility.Tmp      ( /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/Utility/Tmp.hs, /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/dist/setup/Utility/Tmp.o )
[ 7 of 27] Compiling Utility.Env      ( /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/Utility/Env.hs, /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/dist/setup/Utility/Env.o )
[ 8 of 27] Compiling Utility.UserInfo ( /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/Utility/UserInfo.hs, /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/dist/setup/Utility/UserInfo.o )
[ 9 of 27] Compiling Utility.OSX      ( /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/Utility/OSX.hs, /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/dist/setup/Utility/OSX.o )
[10 of 27] Compiling Utility.Monad    ( /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/Utility/Monad.hs, /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/dist/setup/Utility/Monad.o )
[11 of 27] Compiling Utility.Misc     ( /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/Utility/Misc.hs, /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/dist/setup/Utility/Misc.o )
[12 of 27] Compiling Utility.Process  ( /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/Utility/Process.hs, /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/dist/setup/Utility/Process.o )
[13 of 27] Compiling Utility.Path     ( /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/Utility/Path.hs, /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/dist/setup/Utility/Path.o )
[14 of 27] Compiling Utility.FreeDesktop ( /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/Utility/FreeDesktop.hs, /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/dist/setup/Utility/FreeDesktop.o )
[15 of 27] Compiling Assistant.Install.AutoStart ( /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/Assistant/Install/AutoStart.hs, /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/dist/setup/Assistant/Install/AutoStart.o )
[16 of 27] Compiling Utility.SafeCommand ( /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/Utility/SafeCommand.hs, /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/dist/setup/Utility/SafeCommand.o )
[17 of 27] Compiling Utility.ExternalSHA ( /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/Utility/ExternalSHA.hs, /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/dist/setup/Utility/ExternalSHA.o )
[18 of 27] Compiling Utility.Directory ( /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/Utility/Directory.hs, /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/dist/setup/Utility/Directory.o )
[19 of 27] Compiling Common           ( /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/Common.hs, /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/dist/setup/Common.o )
[20 of 27] Compiling Git.Version      ( /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/Git/Version.hs, /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/dist/setup/Git/Version.o )
[21 of 27] Compiling Config.Files     ( /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/Config/Files.hs, /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/dist/setup/Config/Files.o )
[22 of 27] Compiling Assistant.Install.Menu ( /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/Assistant/Install/Menu.hs, /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/dist/setup/Assistant/Install/Menu.o )
[23 of 27] Compiling Build.TestConfig ( /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/Build/TestConfig.hs, /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/dist/setup/Build/TestConfig.o )
[24 of 27] Compiling Build.Version    ( /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/Build/Version.hs, /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/dist/setup/Build/Version.o )
[25 of 27] Compiling Build.Configure  ( /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/Build/Configure.hs, /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/dist/setup/Build/Configure.o )
[26 of 27] Compiling Build.DesktopFile ( /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/Build/DesktopFile.hs, /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/dist/setup/Build/DesktopFile.o )
[27 of 27] Compiling Main             ( /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/Setup.hs, /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/dist/setup/Main.o )
Linking /tmp/git-annex-5.20140108-30015/git-annex-5.20140108/dist/setup/setup ...
  checking version...fatal: Not a git repository (or any of the parent directories): .git
 5.20140107
  checking UPGRADE_LOCATION... not available
  checking git... yes
  checking git version... 1.7.2.5
  checking cp -a... yes
  checking cp -p... yes
  checking cp --reflink=auto... yes
  checking xargs -0... yes
  checking rsync... yes
  checking curl... no
  checking wget... yes
  checking bup... no
  checking quvi... no
  checking newquvi... no
  checking nice... yes
  checking ionice... yes
  checking nocache... no
  checking gpg... gpg
  checking lsof... lsof
  checking git-remote-gcrypt... not available
  checking ssh connection caching... no
  checking sha1... sha1sum
  checking sha256... sha256sum
  checking sha512... sha512sum
  checking sha224... sha224sum
  checking sha384... sha384sum
Configuring git-annex-5.20140108...
Building git-annex-5.20140108...
Preprocessing executable 'git-annex' for git-annex-5.20140108...

Remote/External.hs:29:8:
    Could not find module `Control.Concurrent.STM'
    It is a member of the hidden package `stm-2.4.2'.
    Perhaps you need to add `stm' to the build-depends in your .cabal file.
    Use -v to see a list of the files searched for.
Failed to install git-annex-5.20140108
cabal: Error: some packages failed to install:
git-annex-5.20140108 failed during the building phase. The exception was:
ExitFailure 1

# End of transcript or log.
"""]]

> [[fixed|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnlotDRSLW2JVXY3SLSwhrcHteqUHhTtoY"
 nickname="Péter"
 subject="Not fixed."
 date="2014-01-27T00:43:45Z"
 content="""
It's still failing for me, on both i386 and amd64.

Where exactly is it fixed?

How is it possible to install git-annex using Cabal?

http://stackoverflow.com/questions/21371272/cabal-missing-dependency-error

$ cabal install git-annex --force-reinstalls --bindir=$HOME/bin.local -f\"-assistant -webapp -webdav -pairing -xmpp -dns\"
...
Configuring git-annex-5.20140116...
Building git-annex-5.20140116...
Preprocessing executable 'git-annex' for git-annex-5.20140116...

Remote/External.hs:29:8:
    Could not find module `Control.Concurrent.STM'
    It is a member of the hidden package `stm-2.4.2'.
    Perhaps you need to add `stm' to the build-depends in your .cabal file.
    It is a member of the hidden package `stm-2.2.0.1'.
    Perhaps you need to add `stm' to the build-depends in your .cabal file.
    Use -v to see a list of the files searched for.
cabal: Error: some packages failed to install:
git-annex-5.20140116 failed during the building phase. The exception was: ExitFailure 1
$ cabal install stm --force-reinstalls --bindir=$HOME/bin.local
Resolving dependencies...
All the requested packages are already installed:
stm-2.4.2
Use --reinstall if you want to reinstall anyway.

"""]]


code:
### Please describe the problem.
I didn't spot this  bugs  page before so here is my report I  have  commented on android page

In addition to two existing repositories (1 local /sdcard/annex, which is also avail at/storage/sdcard0/annex + 1 remote) I have added one more local (and said to keep it in sync with original local). But it didn't work -- it "Synced with onerussian.com_annex but not with Annex" and claimed that the /external/extSdCard/Annex doesn't exist, although it is there (and with .git generated etc). When I restarted the deamon I got into a "new" Repository: /storage/extSdCard/Annex which also listed the 1st local but with "Failed to sync with localhost" message -- no remote one listed. Whenever I try to "Switch repository" to /sdcard/annex (the original local) -- it starts loading a new page but gets stuck right there. The only way to revive webui is to go back to Dashboard. Log there says (retyping from the screen so typos might be there):

error: cannot run git-receive-pack '/storage/sdcard0/annex': No such file or directory fatal: unable to fork
### What steps will reproduce the problem?

### What version of git-annex are you using? On what operating system?
android

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

# End of transcript or log.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnRai_qFYPVvEgC6i1nlM1bh-C__jbhqS0"
 nickname="Matthew"
 subject="comment 1"
 date="2013-07-28T11:21:42Z"
 content="""
I can confirm it's virtually impossible to change repository on Android (Stock Galaxy Nexus GSM). you click on the files link (which is hard to see on phone as it is in the notification blocks) and then click click \"Switch repository\", on the next page, select a different one. I think it does actually change, occasionally, you have to kill and restart the daemon at least and I think you also have to restart the phone.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnRai_qFYPVvEgC6i1nlM1bh-C__jbhqS0"
 nickname="Matthew"
 subject="comment 2"
 date="2013-07-28T11:46:30Z"
 content="""
For notification
"""]]


code:
[[!comment format=mdwn
 username="yasin.zaehringer"
 ip="90.218.200.128"
 subject="comment 3"
 date="2014-04-02T11:43:52Z"
 content="""
The bug still exists. It is not possible to change the repository in the WebApp.
"""]]


code:
in a setup where an rsync(+gnupg) remote is shared among different users of the same git-annex repository (ie, the people copying to there use different accounts on the rsync server), acls are not honored under some circumstances.

the error message reads as follows:

    copy …filename… (to prometheus...) Reading passphrase from file descriptor 11

    sending incremental file list
    rsync: recv_generator: mkdir "/home/shared/photos/encrypted_storage/9a6/0ff" failed: Permission denied (13)
    *** Skipping any contents from this failed directory ***
    9a6/0ff/
    rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1070) [sender=3.0.8]

    sent 185 bytes  received 18 bytes  135.33 bytes/sec
    total size is 2119419  speedup is 10440.49

      This could have failed because --fast is enabled.
    failed

the acl used in my particular case is:

    # file: .
    # owner: chrysn
    # group: chrysn
    user::rwx
    group::rwx
    group:family:rwx
    mask::rwx
    other::r-x
    default:user::rwx
    default:group::rwx
    default:group:family:rwx
    default:mask::rwx
    default:other::r-x

sub-directories are observed to have diverging permissions, though:

    # file: 794
    # owner: chrysn
    # group: chrysn
    user::rwx
    group::rwx                      #effective:r-x
    group:family:rwx                #effective:r-x
    mask::r-x
    other::r-x
    default:user::rwx
    default:group::rwx
    default:group:family:rwx
    default:mask::rwx
    default:other::r-x

something seems to apply the umask (default 022) and revoke group write access from the files, overruling the acl. this is not what a umask is normally used for, and smells of [coreutils slavishly observing posix specs that don't consider all features](http://savannah.gnu.org/bugs/?19546) -- the observed effect is exactly what's described there.

the git annex version used is 3.20121017 as in debian, the receiving site uses rsync 3.0.7; the affected directories come from a time when these very versions are known to have been used.

this is probably not a bug of git-annex alone, but affects its operation and might be solvable by invoking rsync differently.

(this is kind of a follow-up on [[forum/__34__permission_denied__34___in_fsck_on_shared_repo]])

[[!tag forwarded]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.27"
 subject="comment 1"
 date="2012-11-06T20:49:47Z"
 content="""
I don't know much about ACLs, but you might try setting `remote.prometheus.annex-rsync-options` with rsync options such as --acls or --no-acls
"""]]


code:
[[!comment format=mdwn
 username="http://christian.amsuess.com/chrysn"
 nickname="chrysn"
 subject="further tracking"
 date="2012-11-11T01:15:41Z"
 content="""
the behavior is rooted in rsync, which behaves similar to `mkdir -p`. it can probably be worked around by configuring the rsync option `--chmod=775`, but that would add executability to files.

i've opened a bug in rsync's bug tracker at <https://bugzilla.samba.org/show_bug.cgi?id=9377>, let's see what the developer says.
"""]]


code:
Creating additional branches in history seems to slow down the 'git annex unused' command quadratically, even if the location of the branches should be irrelevant as far as unused data goes.

This was tested on:

	$ git annex version
	git-annex version: 3.20130216
	local repository version: 3
	default repository version: 3
	supported repository versions: 3
	upgrade supported from repository versions: 0 1 2

What steps will reproduce the problem?

	$ mkdir a
	$ cd a
	$ git init
	$ git annex init
	$ i=0 ; while test $i -lt 1000; do dd if=/dev/urandom of=$i.img bs=1M count=1; i=$(($i+1)); done
	$ git annex add .
	$ git commit -m"foo"
	$ git rm 1*
	$ git commit -m"bar"
	$ git log --oneline --decorate
	ffcca3a (HEAD, master) bar
	3e7793d foo
	$ time -p git annex unused
	unused . (checking for unused data...) (checking master...)
	(...)
	real 0.76
	user 0.40
	sys 0.06
	git commit --allow-empty -m"baz"
	$ git log --oneline --decorate
	4390c32 (HEAD, master) baz
	ffcca3a bar
	3e7793d foo
	$ time -p git annex unused
	unused . (checking for unused data...) (checking master...)
	(...)
	real 0.75
	user 0.38
	sys 0.07
	$ git branch boo HEAD^
	$ time -p git annex unused
	unused . (checking for unused data...) (checking boo...) (checking master...)
	(...)
	real 1.29
	user 0.62
	sys 0.08
	arand@mas:~/tmp/more/a(master)$ git branch beeboo HEAD^
	4390c32 (HEAD, master) baz
	ffcca3a (boo, beeboo) bar
	3e7793d foo
	arand@mas:~/tmp/more/a(master)$ time -p git annex unused
	unused . (checking for unused data...) (checking beeboo...) (checking master...)
	(...)
	real 2.50
	user 1.12
	sys 0.14
	$ git branch -d boo beeboo
	$ git log --oneline --decorate
	4390c32 (HEAD, master) baz
	ffcca3a bar
	3e7793d foo
	$ time -p git annex unused
	unused . (checking for unused data...) (checking master...)
	(...)
	real 0.77
	user 0.42
	sys 0.04

What is the expected output? What do you see instead?

I would expect the time to be the same in all the above cases.

What version of git-annex are you using? On what operating system?

	$ git annex version
	git-annex version: 3.20130216

On current Debian sid/experimental

> [[Done]], thanks to guilhem. We ended up using a different algorythm
> which is faster yet, basically it now does a diff-index between the
> index and each branch for its second stage bloom filter.
> Speedup is 30x with 0 (or 1?) branch, and then massive for each
> additional branch. --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.210"
 subject="comment 1"
 date="2013-02-26T20:29:05Z"
 content="""
`git annex unused` finds content that is not used by the working tree or by *any* branch is unused. For the working tree, it can look at the symlinks on disk, which is the fastest option.

For branches, it has to first use `git ls-tree` to find the files on the branch, and then use `git cat-file` to look up the key used by each file. It does this as fast as it can (eg, it runs a single `git cat-file --batch`, rather than one process per file). Still, this is pulling potentially a lot of data out of git, and it gets pretty slow.

I've spent a lot of time optimising this as much as is possible with these constraints. One nice one is that, if it finds no unused keys after checking the working tree, it can stop, rather than checking any branches. Your example avoids this optimisation.

Another optimisation is to only check each git ref once, even if multiple branches refer to it. You can see this optmisation firing in your transcript, when it only shows it's checking one branch of the two identical branches you've made.

Indeed, if you go on and add 100 identical branches, you'll find it runs in just about the same time it ran with 2 branches. (There's a little overhead in getting the list of branches and throwing out the duplicates, but that's all.)

What then explains your numbers? Well, I have no idea. I cannot replicate them; I tend to see about the same amount of time taken with two duplicate branches as with one branch. I suspect you just didn't get statistically valid results, which playing around with `time` at the command line often doesn't,
due to caching, other active processes, etc.
"""]]


code:
[[!comment format=mdwn
 username="https://launchpad.net/~arand"
 nickname="arand"
 subject="comment 2"
 date="2013-02-27T15:53:37Z"
 content="""
Hmm, indeed, after further testing it seems like the increased time due to the duplicate branch seems to have been a random quirk, bleh :(

But shouldn't it theoretically be possible to optimize out much of the overhead of multiple very-similar (though not identical) branches though?

I've experimented around with ls-tree and cat-file in a bash script[1], and in this primitive implementation the running time seems to be considerably lower (~0.3s vs ~4s), with much less overhead for extra very-similar branches (~0.7s vs ~37s)

Am I missing some key element that's the reason for the time taken by git annex unused?

[1] primitive annex unused script: [https://gitorious.org/arand-scripts/arand-scripts/blobs/master/annex-funused](https://gitorious.org/arand-scripts/arand-scripts/blobs/master/annex-funused)

timing script: [https://gitorious.org/arand-scripts/arand-scripts/blobs/master/annex-testunused](https://gitorious.org/arand-scripts/arand-scripts/blobs/master/annex-testunused)

"""]]


code:
[[!comment format=mdwn
 username="arand"
 ip="130.243.226.21"
 subject="comment 3"
 date="2013-08-10T17:00:21Z"
 content="""
So, if I've understood it correctly (please correct me if that's not the case :) )

Currently git-annex unused goes through this process

* Look through all files in the index and find those which are git-annex keys (git ls-tree + git cat-file)
* Look through all files the current ref and find those which are git-annex keys (git ls-tree + git cat-file)
* For each ref in the repo
  - Look through all files and find those which are git-annex keys (git ls-tree + git cat-file)
* Then at the end
  - Compare this list of keys with what is stored in .git/annex/objects
  - Print out any objects which does not match a key.

If that's the case, it means if that if you have multiple refs, even is they only differ by single empty commits, git-annex will end up doing a cat-file for the same file multiple times (one per ref), which is expensive.

Would it be possible to change the algorithm for git-annex unused into instead something like:

* For the index, HEAD, and all refs
  - Create a list all files and remove those which are duplicates based on their sha1 hash (git ls-tree | uniq)
* Then Look through this reduced list to find those which are git-annex keys (git cat-file)
* Then check as before

Unless this bypasses some safety or case I've overlooked, I think it should be possible to speed up git-annex unused quite a bit.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4978:f:21a::2"
 subject="comment 4"
 date="2013-08-11T12:48:47Z"
 content="""
I think that could work. It would probably tend to use more memory than the current method, but only a small constant multiplier more. And unused is already the one command that necessarily needs to hold information about the whole repository in memory.

Note that git cat-file is only needed when dealing with branches other than the current working tree. In that special case, it can, and AFAIK does have the optimisation of looking directly at the symlink target instead. Your method may turn out to be both slower and use more memory in that case. It may make sense to special case handling of that case and keep the current code paths there (most of the necessary code for it is used by other stuff anyway).
"""]]


code:
[[!comment format=mdwn
 username="arand"
 ip="130.243.226.21"
 subject="comment 5"
 date="2013-08-11T20:43:22Z"
 content="""
I've compared my bash/coreutils implementation mentioned above [annex-funused](https://gitorious.org/arand-scripts/arand-scripts/blobs/918cc79b99e22cbdca01ea4de10e2ca64abfc27a/annex-funused) with `git annex unused` in various situations, and from what I've seen `annex-funused` is pretty much always faster.

In the case of no unused files they seem to be about the same.

In all other cases there is a very considerable difference, for example, in my current main annex I get:

    $ time git annex unused
    unused . (checking for unused data...) (checking master...) (checking synced/master...) (checking barracuda160G/master...) ok

    real    5m13.830s
    user    2m0.444s
    sys     0m28.344s

whereas

    $ time annex-funused
     == WARNING ==
    This program should NOT be trusted to reliably find unused files in the
    git annex.

    real    0m1.569s
    user    0m2.024s
    sys     0m0.184s

I tried to check memory usage via `/usr/bin/time -v` as well, and that showed (re-running in the same annex as above)

annex-funused

    Maximum resident set size (kbytes): 13560

git annex unused

    Maximum resident set size (kbytes): 29120

I've also written a comparison script [annex-testunused](https://gitorious.org/arand-scripts/arand-scripts/blobs/918cc79b99e22cbdca01ea4de10e2ca64abfc27a/annex-testunused) (needs annex-funused in $PATH) which creates an annex with a bunch of unused files and compares the running time for both versions:

<pre>
$ annex-testunused
Initialized empty Git repository in /tmp/tmp.fmsAvsPTcd/.git/
init  ok
(Recording state in git...)
###
* b2840d7 (HEAD, master) delete ~1100 files
* c4a1e3a add 3000 files
* bc19777 (git-annex) update
* b3e6539 update
* bec2c8f branch created
annex unused
real 0m4.154s
real 0m2.029s
real 0m2.044s
annex funused
real 0m0.923s
real 0m0.933s
real 0m0.905s
Initialized empty Git repository in /tmp/tmp.7qFoCRWzB3/.git/
init  ok
(Recording state in git...)
###
* a5ff392 (HEAD, master) empty
* cca4810 (1) delete ~1100 files
* 587c406 add 3000 files
* de0afeb (git-annex) update
* 37b7881 update
* 1735062 branch created
annex unused
real 0m3.499s
real 0m3.443s
real 0m3.435s
annex funused
real 0m0.956s
real 0m0.956s
real 0m0.874s
Initialized empty Git repository in /tmp/tmp.L5fjdAgnFv/.git/
init  ok
(Recording state in git...)
###
* 94463a0 (HEAD, master) empty
* e115619 (10) empty
* 20686d4 (9) empty
* 2e01a3f (8) empty
* 043289d (7) empty
* 6a52966 (6) empty
* 0dc866d (5) empty
* 35db331 (4) empty
* 48504bc (3) empty
* e25cac7 (2) empty
* 655d026 (1) delete ~1100 files
* 91a07d1 add 3000 files
* 3c9ac62 (git-annex) update
* c5736e0 update
* 862d5b8 branch created
annex unused
real 0m16.242s
real 0m16.277s
real 0m16.246s
annex funused
real 0m0.960s
real 0m0.960s
real 0m0.927s
</pre>

So, unless I've missed something fundamental (I keep thinking I might have...), it seems to be very consistently faster, and scale ok where `git annex unused` scales rather poorly.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4978:f:21a::2"
 subject="comment 6"
 date="2013-08-12T04:14:18Z"
 content="""
The memory usage is probably lower because `sort` and `comm` and bash's `<(command)` all have particularly well tuned memory usage with 37 years of history behind them. Particularly GNU `sort` will transparently use a temp file rather than storing too much data in memory, and does rather sophisticated stuff to make that work efficiently. It's rather harder to get that kind of behavior when not using the unix tools and instead using stock programming language primatives like sort() and hashes.

I still suspect that `git cat-file` is slower than a direct readlink(2) of the symlink, when that can be done.
"""]]


code:
[[!comment format=mdwn
 username="arand"
 ip="130.243.226.21"
 subject="comment 7"
 date="2013-08-12T18:07:44Z"
 content="""
Hmm, probably, do you think this way of getting everything and then doing the filtering could work reasonably in git-annex? Assuming \"git cat-file --batch\" is the main bottleneck, reducing the amount of runs for it would still likely be an improvement?
"""]]


code:
### Please describe the problem.
I tried to add a remote server to my annex repo using the webapp.

### What steps will reproduce the problem?
Configuration -> Repositories -> Remote server -> Fill in stuff, use public key auth -> click "check this server" -> enjoy

### What version of git-annex are you using? On what operating system?
OpenSUSE Tumbleweed, git-annex 5.20140709-gc75193e

### Please provide any additional information below.
Error message: `user error (gpg ["--batch","--no-tty","--use-agent","--quiet","--trust-model","always","--with-colons","--list-secret-keys","--fixed-list-mode"] exited 2)`

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
[2014-07-11 12:03:52 CEST] main: starting assistant version 5.20140709-gc75193e
[2014-07-11 12:03:53 CEST] UpgradeWatcher: Finished upgrading git-annex to version 5.20140709-gc75193e
(scanning...) [2014-07-11 12:03:53 CEST] Watcher: Performing startup scan
(started...) gpg: /home/zilti/.gnupg/gpg.conf:200: argument not expected
gpg: /home/zilti/.gnupg/gpg.conf:201: invalid option
gpg: /home/zilti/.gnupg/gpg.conf:200: argument not expected
gpg: /home/zilti/.gnupg/gpg.conf:201: invalid option

# End of transcript or log.
"""]]

[[!tag moreinfo]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="obvious questions"
 date="2014-07-11T18:12:44Z"
 content="""
From what source did you install git-annex?

What version of gpg is installed on your system? What is the contents of line 200 and 201 of/home/zilti/.gnupg/gpg.conf?
"""]]


code:
Adding eg a ssh remote when the remote repo already exists, in the webapp,
resets its group to transfer. It also clears any preferred content
settings.

Adding existing local repositories or repositories from removable drives
may have the same problems. Didn't check yet.

[[!tag confirmed]] --[[Joey]]

> Fixed for local repos and repos on removable drives. Still open for
> ssh remotes (incl gcrypt). --[[Joey]]

>> Fixed for ssh (including gcrypt) too.
>>
>> Also affected enabling existing special remotes, like webdav; that's
>> also fixed. [[done]] --[[Joey]]


code:
### Please describe the problem.

"git annex --pathdepth=3 addurl URL" displays this error message and reports "failed", even though the operation succeeds when using --fast:

    git-annex: /media/usb32g-1/annex/musikk/.git/annex/tmp/2325: rename: does not exist (No such file or directory)

### What steps will reproduce the problem?

I have lots of downloaded files I'd like to add an URL to, so I'm doing this:

    $ ga addurl --fast --pathdepth=3 $(for f in *.mp3; do echo http://traffic.libsyn.com/geologicpodcast/$f; done)
    addurl
    git-annex: /media/usb32g-1/annex/musikk/.git/annex/tmp/2325: rename: does not exist (No such file or directory)
    failed
    addurl
    git-annex: /media/usb32g-1/annex/musikk/.git/annex/tmp/2325: rename: does not exist (No such file or directory)
    failed
    addurl
    git-annex: /media/usb32g-1/annex/musikk/.git/annex/tmp/2325: rename: does not exist (No such file or directory)
    [... Delete 166 lines ...]
    addurl
    git-annex: /media/usb32g-1/annex/musikk/.git/annex/tmp/2325: rename: does not exist (No such file or directory)
    failed
    addurl
    git-annex: /media/usb32g-1/annex/musikk/.git/annex/tmp/2325: rename: does not exist (No such file or directory)
    failed
    (Recording state in git...)
    git-annex: addurl: 60 failed
    $

This "ga" command is a wrapper around git-annex to make it work everywhere with the .tar.gz version:

    #!/bin/bash

    #=======================================================================
    # ga
    # File ID: e89047ce-29d1-11e2-bb6f-00c0a8deee11
    #=======================================================================

    if test "$1" = "sync"; then
        stat_output="$(git status --porcelain | grep -v '^??')"
        test -n "$stat_output" && { echo ga: Repo has modifications, will not git-annex sync >&2; exit 1; }
    fi
    /opt/git-annex/runshell /opt/git-annex/git-annex "$@"

$PATH is also set up to include /opt/git-annex/bin/ , and all other operations work nicely. In my ~/bin/ directory I've also symlinked this script to "git-annex".

When dropping "--fast", this happens:

    $ ga addurl --pathdepth=3 $(for f in *.mp3; do echo http://traffic.libsyn.com/geologicpodcast/$f; done | tail -20 | head -1)
    addurl  (downloading http://traffic.libsyn.com/geologicpodcast/GeologicPodcast302-Feb28-13.mp3 ...) --2013-07-04 14:08:01--  http://traffic.libsyn.com/geologicpodcast/GeologicPodcast302-Feb28-13.mp3
    Resolving traffic.libsyn.com (traffic.libsyn.com)... 204.16.245.39
    Connecting to traffic.libsyn.com (traffic.libsyn.com)|204.16.245.39|:80... connected.
    HTTP request sent, awaiting response... 302 Found
    Location: http://ec.libsyn.com/p/6/9/b/69b9837556ed5238/GeologicPodcast302-Feb28-13.mp3?d13a76d516d9dec20c3d276ce028ed5089ab1ce3dae902ea1d01cf853ed5cd5c0f85&c_id=5440885 [following]
    --2013-07-04 14:08:02--  http://ec.libsyn.com/p/6/9/b/69b9837556ed5238/GeologicPodcast302-Feb28-13.mp3?d13a76d516d9dec20c3d276ce028ed5089ab1ce3dae902ea1d01cf853ed5cd5c0f85&c_id=5440885
    Resolving ec.libsyn.com (ec.libsyn.com)... 68.232.34.133
    Connecting to ec.libsyn.com (ec.libsyn.com)|68.232.34.133|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 53173384 (51M) [audio/mpeg]
    Saving to: `/media/usb32g-1/annex/musikk/.git/annex/tmp/URL--http&c%%traffic.libsyn.com%geologicpodcast%GeologicPodcast302-Feb28-13.mp3'

    100%[==================================================================================================================================================================>] 53,173,384  3.50M/s   in 45s

    2013-07-04 14:08:47 (1.13 MB/s) - `/media/usb32g-1/annex/musikk/.git/annex/tmp/URL--http&c%%traffic.libsyn.com%geologicpodcast%GeologicPodcast302-Feb28-13.mp3' saved [53173384/53173384]

    (checksum...)
    git-annex: /media/usb32g-1/annex/musikk/.git/annex/tmp/3970: rename: does not exist (No such file or directory)
    failed
    (Recording state in git...)
    git-annex: addurl: 1 failed
    $

git-annex now believes that the file is stored locally, but an fsck shows how things really are:

    $ ga fsck
    fsck GeologicPodcast02-Feb21-07.mp3 ok
    fsck GeologicPodcast03-Feb28-07.mp3 ok
    [... Delete 35 lines ...]
    fsck GeologicPodcast301-Feb22-13.mp3 ok
    fsck GeologicPodcast302-Feb28-13.mp3 (fixing location log)
      ** Based on the location log, GeologicPodcast302-Feb28-13.mp3
      ** was expected to be present, but its content is missing.
    failed
    fsck GeologicPodcast303-Mar07-13.mp3 ok
    fsck GeologicPodcast303.1_Fancast6-Mar09-13.mp3 ok
    [... Delete 16 lines ...]
    fsck GeologicPodcast72-Jul03-08.mp3 ok
    (Recording state in git...)
    git-annex: fsck: 3 failed
    $

After this fsck, everything is OK again; the location log is correct and "get" from this URL works.

The file data didn't exist locally (it's stored on another disk) when doing this, but I also experimented what would happen if the file data did exist:

    $ rm GeologicPodcast297-Jan24-13.mp3
    $ wget http://traffic.libsyn.com/geologicpodcast/GeologicPodcast297-Jan24-13.mp3
    --2013-07-04 14:35:15--  http://traffic.libsyn.com/geologicpodcast/GeologicPodcast297-Jan24-13.mp3
    Resolving traffic.libsyn.com (traffic.libsyn.com)... 204.16.245.39
    Connecting to traffic.libsyn.com (traffic.libsyn.com)|204.16.245.39|:80... connected.
    HTTP request sent, awaiting response... 302 Found
    Location: http://hwcdn.libsyn.com/p/9/d/f/9df13b1a6d622c46/GeologicPodcast297-Jan24-13.mp3?c_id=5341445&expiration=1372945406&hwt=476d03abba40d71119011bf7cb51f68a [following]
    --2013-07-04 14:35:16--  http://hwcdn.libsyn.com/p/9/d/f/9df13b1a6d622c46/GeologicPodcast297-Jan24-13.mp3?c_id=5341445&expiration=1372945406&hwt=476d03abba40d71119011bf7cb51f68a
    Resolving hwcdn.libsyn.com (hwcdn.libsyn.com)... 69.16.175.42, 69.16.175.10
    Connecting to hwcdn.libsyn.com (hwcdn.libsyn.com)|69.16.175.42|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 49299974 (47M) [audio/mpeg]
    Saving to: ‘GeologicPodcast297-Jan24-13.mp3’

    100%[===========================================================================================================================================>] 49,299,974  5.88MB/s   in 30s

    $ ga add
    add GeologicPodcast297-Jan24-13.mp3 (checksum...) ok
    (Recording state in git...)
    $ ga addurl --pathdepth=3 http://traffic.libsyn.com/geologicpodcast/GeologicPodcast297-Jan24-13.mp3

At this stage, the data for this .mp3 file exist locally, but git-annex is either unaware of it, or it insists on verifying the data. Anyhow, it downloads the file:

    $ ga addurl --pathdepth=3 http://traffic.libsyn.com/geologicpodcast/GeologicPodcast297-Jan24-13.mp3
    addurl  (downloading http://traffic.libsyn.com/geologicpodcast/GeologicPodcast297-Jan24-13.mp3 ...) --2013-07-04 14:42:40--  http://traffic.libsyn.com/geologicpodcast/GeologicPodcast297-Jan24-13.mp3
    Resolving traffic.libsyn.com (traffic.libsyn.com)... 204.16.245.39
    Connecting to traffic.libsyn.com (traffic.libsyn.com)|204.16.245.39|:80... connected.
    HTTP request sent, awaiting response... 302 Found
    Location: http://ec.libsyn.com/p/9/d/f/9df13b1a6d622c46/GeologicPodcast297-Jan24-13.mp3?d13a76d516d9dec20c3d276ce028ed5089ab1ce3dae902ea1d01cf853ed5cc54b707&c_id=5341445 [following]
    --2013-07-04 14:42:41--  http://ec.libsyn.com/p/9/d/f/9df13b1a6d622c46/GeologicPodcast297-Jan24-13.mp3?d13a76d516d9dec20c3d276ce028ed5089ab1ce3dae902ea1d01cf853ed5cc54b707&c_id=5341445
    Resolving ec.libsyn.com (ec.libsyn.com)... 68.232.34.133
    Connecting to ec.libsyn.com (ec.libsyn.com)|68.232.34.133|:80... connected.
    HTTP request sent, awaiting response... 206 Partial Content
    Length: 49299974 (47M), 48727654 (46M) remaining [audio/mpeg]
    Saving to: `/media/usb32g-1/annex/musikk/.git/annex/tmp/URL--http&c%%traffic.libsyn.com%geologicpodcast%GeologicPodcast297-Jan24-13.mp3'

    100%[====================================================================================================================================================================>] 49,299,974  2.07M/s   in 88s

    2013-07-04 14:44:10 (538 KB/s) - `/media/usb32g-1/annex/musikk/.git/annex/tmp/URL--http&c%%traffic.libsyn.com%geologicpodcast%GeologicPodcast297-Jan24-13.mp3' saved [49299974/49299974]

    (checksum...)
    git-annex: /media/usb32g-1/annex/musikk/.git/annex/objects/gp/7Z/SHA256-s49299974--ff6758202d7168a0548b004d0c6e79a095b7738f442aa14d2603665914bee7e0/SHA256-s49299974--ff6758202d7168a0548b004d0c6e79a095b7738f442aa14d2603665914bee7e0: rename: does not exist (No such file or directory)
    failed
    (Recording state in git...)
    git-annex: addurl: 1 failed
    2013-07-04 14:44:11 sunny@dg-vbox:/media/usb32g-1/annex/musikk/podcast/Geologic Podcast (master)
    $

The error message appears again, but because the data did exist from before, git-annex fsck has nothing to complain about:

    $ ga fsck
    [...]
    fsck GeologicPodcast297-Jan24-13.mp3 (checksum...) ok
    [...]
    2013-07-04 14:44:33 sunny@dg-vbox:/media/usb32g-1/annex/musikk/podcast/Geologic Podcast (master)
    $

### What version of git-annex are you using? On what operating system?

v4.20130627 Linux i386 and AMD64. Both versions behave the same way. Tested on two computers:

Computer #1 (AMD64 version):

    $ uname -a
    Linux dg-vbox 3.8.0-19-generic #30-Ubuntu SMP Wed May 1 16:35:23 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
    $ lsb_release -a
    No LSB modules are available.
    Distributor ID: LinuxMint
    Description:    Linux Mint 15 Olivia
    Release:        15
    Codename:       olivia

Computer #2 (i386 version):

    $ uname -a
    Linux linode 2.6.39.1-linode34 #1 SMP Tue Jun 21 10:29:24 EDT 2011 i686 GNU/Linux
    $ lsb_release -a
    Distributor ID: Ubuntu
    Description:    Ubuntu 10.04.4 LTS
    Release:        10.04
    Codename:       lucid
    No LSB modules are available.

Also tested with v4.20130601, same thing happens.

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

All interesting transcripts are posted above with cruft removed.

# End of transcript or log.
"""]]

> The cause of this bug is using --pathdepth=3 when the url
> only has two components in its path. So, kinda GIGO, but
> let's fix this anyway. ;)
>
> [[fixed|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="http://sunny256.sunbase.org/"
 nickname="sunny256"
 subject="comment 1"
 date="2013-07-08T19:56:58Z"
 content="""
Aha, in other words, some kind of PEBCAC put in a polite way. :) I thought this should be correct, as the man page says

> For example, --pathdepth=1 will use \"dir/subdir/bigfile\", while --pathdepth=3 will use \"bigfile\".

This is how I thought it worked:

* 0 = hostname, path, filename
* 1 = path, filename
* 2 = filename

After reading your answer I first thought it was a typo in the documentation, but of course the number specify the level of subdirectories to use in the file name. D'oh!

Thanks for the fix and information.
"""]]


code:
### Please describe the problem.
The app fails to open after the install

### What steps will reproduce the problem?
Downloaded, installed, chose open.

### What version of git-annex are you using? On what operating system?
today's version. android 4.2.1

### Please provide any additional information below.

I also sent this by email.

Falling back to hardcoded app location; cannot find expected files in /data/app-lib
git annex webapp
u0_a72@android:/sdcard/git-annex.home $ git annex webapp
Launching web browser on http://127.0.0.1:48812/?auth=cf45448c3690730f05f1a9567e62c6a3cf8d25c43ed14362c8ae78601de0e96d32d2b02923ba95962e80c9bd8ffab621918dd582741f18160f6e2565af61aba5
Starting: Intent { act=android.intent.action.VIEW dat=http://127.0.0.1:48812/?auth=cf45448c3690730f05f1a9567e62c6a3cf8d25c43ed14362c8ae78601de0e96d32d2b02923ba95962e80c9bd8ffab621918dd582741f18160f6e2565af61aba5 }
java.lang.SecurityException: Permission Denial: startActivity asks to run as user -2 but is calling from user 0; this requires android.permission.INTERACT_ACROSS_USERS_FULL
        at android.os.Parcel.readException(Parcel.java:1425)
        at android.os.Parcel.readException(Parcel.java:1379)
        at android.app.ActivityManagerProxy.startActivityAsUser(ActivityManagerNative.java:1906)
        at com.android.commands.am.Am.runStart(Am.java:494)
        at com.android.commands.am.Am.run(Am.java:109)
        at com.android.commands.am.Am.main(Am.java:82)
        at com.android.internal.os.RuntimeInit.nativeFinishInit(Native Method)
        at com.android.internal.os.RuntimeInit.main(RuntimeInit.java:235)
        at dalvik.system.NativeStart.main(Native Method)
failed to start web browser

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/debug.log

# End of transcript or log.
"""]]

This is a duplicate of [[Android_app_permission_denial_on_startup]]

[[closing|done]] as dup. --[[Joey]]


code:
### Please describe the problem.
Impossible installation on Android 4.3

### What version of git-annex are you using? On what operating system?
The lastest version of git-annex, and Android 4.3, **without sdcard** (Wiko Wax)

### Please provide any additional information below.

The message given by git-annex:

    Falling back to hardcoded app location; cannot find expected files in /data/app-lib
    mkdir: can't create directory '/sdcard/git-annex.home': Permission denied
    mkdir of /sdcard/git-annex.home failed !
    lib/lib.runshell.so: line 133: can't create /sdcard/git-annex.home/git-annex-install.log: Permission denied
    Installation failed ! Please report a but and attach /sdcard/git-annex.home/git-annex-install.log

[[!meta title="android 4.3 install failed on android device without sdcard"]]
[[!tag moreinfo]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="24.159.78.125"
 subject="comment 1"
 date="2014-06-02T16:28:42Z"
 content="""
git-annex needs a directory to use to store its repositories. If your android device does not have a /sdcard directory (which does not have to be a literal sd card, but just someplace you can write to), does it have any equivilant directory?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkPSASwemBzccJmjkotESlbUSs5GPFiPCs"
 nickname="Lin"
 subject="Ok, but ?"
 date="2014-06-02T18:22:24Z"
 content="""
You mean that I have to create the missed files ?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 3"
 date="2014-06-04T18:08:39Z"
 content="""
I asked if your Android device has an eqivilant directory to /sdcard that you can write to. If so, I can fix git-annex to use that directory. But you need to tell me what it is first!
"""]]


code:
The android build is broken. Annoying evilsplicer failure in the new
bootstrap3 code. --[[Joey]]

[[fixed|done]]


code:
Just starting using v3, even more awesome, thanks!

With git-annex, I take the habit to do copies of files without restriction, as they end up into (cheap) symlink copies.
However, if 2 copies are unannexed, only one is restored, the other becomes a broken symlink, so I kind of loose some information
(my use case: I have a repo on which I recently started using annex, but most of the files, which i would want to be annexed, are only in git,
so my plan is to unninit this repo, delete the .git dir, and then annex everything, as I don't mind the history).

Rafaël

> The only way for git-annex to support this in its current state would be
> for the unannex command to copy the file content from the annex, rather
> than moving it out. Then multiple links to the same content could be
> unannexed.
>
> But, this would be slower, and would depend on a later `unused` and
> `dropunused` to actually remove the content. While doable, by use case
> for unannex is more to quickly undo a mistaken add, and it's unlikely there
> are multiple symlinks to the same content in this situation. --[[Joey]]

[[!tag done]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnpdM9F8VbtQ_H5PaPMpGSxPe_d5L1eJ6w"
 nickname="Rafaël"
 subject="comment 1"
 date="2011-07-04T16:57:25Z"
 content="""
You convince me for unannex, but isn't the goal of uninit to revert all annex operations? In the current state, a clean revert is not possible (because of the broken symlinks after uninit). Instead of copying, using hard links is out of question?

For my needs, is the command \"git annex unlock .\" (from the root of the repo) a correct workaround?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmJfIszzreLNvCqzqzvTayA9_9L6gb9RtY"
 nickname="Joey"
 subject="comment 2"
 date="2011-07-04T20:25:38Z"
 content="""
Indeed, uninit needed to be improved. I've done so. Also, unannex --fast can be used to make hard links to content left in the annex.
"""]]


code:
### Please describe the problem.
missing libselinux.so.1 ?

### What steps will reproduce the problem?
[[!format sh """
./runshell
cd ~
$ git clone ssh://greg@host/path/to/annex/of/Photos Photos
Cloning into 'Photos'...
ssh: error while loading shared libraries: libselinux.so.1: cannot open shared object file: No such file or directory
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
"""]]

same command (copy/paste) works on my laptop running 4.20131106

### What version of git-annex are you using? On what operating system?
[[!format sh """
$ git annex version
git-annex version: 5.20131216-g604e6b8
build flags: Assistant Pairing S3 Inotify DBus XMPP Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web glacier hook
"""]]

arm build (downloaded from [[install/Linux_standalone]])

> ssh was not shimmed. [[fixed|done]]; build will update in a few hours
> --[[Joey]]


code:
[[!comment format=mdwn
 username="http://grossmeier.net/"
 nickname="greg"
 subject="comment 1"
 date="2013-12-19T19:05:04Z"
 content="""
Just tried again (redownloaded the build at ~11:50am Pacific on Dec 19th), same behavior:

[[!format sh \"\"\"
$ git-annex version
git-annex version: 5.20131218-gc5767e2
build flags: Assistant Pairing S3 Inotify DBus XMPP Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web glacier hook
$ git clone greg@host:/path/to/Photos Photos
Cloning into 'Photos'...
ssh: error while loading shared libraries: libselinux.so.1: cannot open shared object file: No such file or directory
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
$ which ssh
/volume1/downloads/git-annex.linux/bin/ssh
$ ldd /volume1/downloads/git-annex.linux/bin/ssh
/volume1/downloads/git-annex.linux/bin/ssh: /lib/libc.so.6: version `GLIBC_2.15' not found (required by /volume1/downloads/git-annex.linux/bin/ssh)
/volume1/downloads/git-annex.linux/bin/ssh: /lib/libc.so.6: version `GLIBC_2.8' not found (required by /volume1/downloads/git-annex.linux/bin/ssh)
/volume1/downloads/git-annex.linux/bin/ssh: /lib/libc.so.6: version `GLIBC_2.17' not found (required by /volume1/downloads/git-annex.linux/bin/ssh)
/volume1/downloads/git-annex.linux/bin/ssh: /lib/libcrypto.so.1.0.0: no version information available (required by /volume1/downloads/git-annex.linux/bin/ssh)
/volume1/downloads/git-annex.linux/bin/ssh: /lib/libcrypto.so.1.0.0: no version information available (required by /volume1/downloads/git-annex.linux/bin/ssh)
	libselinux.so.1 => not found
	libcrypto.so.1.0.0 => /lib/libcrypto.so.1.0.0 (0x40026000)
	libdl.so.2 => /lib/libdl.so.2 (0x401ba000)
	libz.so.1 => /lib/libz.so.1 (0x401c5000)
	libresolv.so.2 => /lib/libresolv.so.2 (0x401e2000)
	libgssapi_krb5.so.2 => /lib/libgssapi_krb5.so.2 (0x401fc000)
	libc.so.6 => /lib/libc.so.6 (0x40235000)
	/lib/ld-linux.so.3 (0x40000000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x40364000)
	libkrb5.so.3 => /lib/libkrb5.so.3 (0x40378000)
	libk5crypto.so.3 => /lib/libk5crypto.so.3 (0x40426000)
	libcom_err.so.3 => /lib/libcom_err.so.3 (0x40458000)
	libkrb5support.so.0 => /lib/libkrb5support.so.0 (0x40462000)
\"\"\"]]
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="comment 2"
 date="2013-12-19T19:55:33Z"
 content="""
Build took longer than I thought, but should finally be done in an hour or two.
"""]]


code:
### Please describe the problem.
When syncing from server A (real server) to server B (a synology NAS) it fails on git-upload-pack and -receive-pack

### What steps will reproduce the problem?
git-annex sync

### What version of git-annex are you using? On what operating system?
on server A: 3.20121017 (yeah, I should update)
on server B: 5.20131220-g73492b2

### Please provide any additional information below.

On server B, after running ./runshell
[[!format sh """
$ git-upload-pack
/volume1/downloads/git-annex.linux/shimmed/sh/sh: 19: git-upload-pack: not found
$ git-receive-pack
/volume1/downloads/git-annex.linux/shimmed/sh/sh: 20: git-receive-pack: not found
"""]]

> [[fixed|done]]; added git, git-upload-pack, and git-receive-pack wrappers
> to git-annex.linux/. Also to equivilant place in OSX app. --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 1"
 date="2013-12-24T20:20:07Z"
 content="""
This has nothing to do with shimming at all AFACS.

runshell does nothing to make git-upload-pack or git-receive-pack be in your PATH. They do not need to be, for `git upload-pack` and `git receive-pack` to work. At least `git-annex-shell`, and probably also `git-shell` both use the latter, rather than the former.

But in the case that a git remote's ssh key is not set up to use git-annex-shell (ie, a git remote not configured using the assistant, or configured using the asistant but with a pre-existing passwordless ssh key), and git-annex was installed on the remote using the git-annex standalone bundle, and there is no other git installation, then yes, git will try to ssh to there and run `git-upload-pack` etc.
"""]]


code:
##What steps will reproduce the problem?

Follow the [share with a friend walkthrough](http://git-annex.branchable.com/assistant/share_with_a_friend_walkthrough/).  I use my Google Talk account to pair with myself.  With the assistant running, log in to Google Talk on the web interface and set your status 'Invisible'.

##What is the expected output? What do you see instead?

I expect to remain invisible, but I get the following warning: "Oops! You are not invisible because you're logged into Google Talk from another client, device or location that doesn't support invisibility."

##What version of git-annex are you using? On what operating system?

4.20130314 on Linux.

##Please provide any additional information below.

Syncing between the repositories works ok.

[[!tag /design/assistant]]


code:
[[!comment format=mdwn
 username="http://smcv.pseudorandom.co.uk/"
 nickname="smcv"
 subject="GTalk wants you to implement its "shared status" protocol"
 date="2013-03-16T11:59:22Z"
 content="""
For invisibility to work, GTalk demands that every connected resource implements its \"shared status\" extension. Given that git-annex sets negative priority, this seems like a GTalk bug - it should ignore negative-priority resources when deciding whether it can be invisible - but it seems unlikely to be fixed.

We implement shared status in telepathy-gabble (mostly in src/conn-presence.c, tests in tests/twisted/presence/shared-status.py) if it's any help.

<http://code.google.com/apis/talk/jep_extensions/shared_status.html>
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 2"
 date="2013-03-16T15:39:14Z"
 content="""
Thanks for the link, that perhaps explains the continuing oddities I've seen with google talk and presence information sometimes not being sent.

It's not clear to me if git-annex could use this extension at all. It seems to say that if I use it, my regular presence messages won't be sent. Which is a problem, since git-annex uses extended content in presence messages to indicate that they're coming from a git-annex client, which is important to make xmpp pairing work. (It's also used to send push notifications, but that's not essential.)

Also, of course, I'm reluctant to implement a non-standard extension that may change at any time, especially when it's attached to a service that currently has the FSF unhappy <http://www.fsf.org/blogs/sysadmin/google-backslides-on-federated-instant-messaging-on-purpose>.

Perhaps the real solution is to make the assistant encourage use of some other XMPP server.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 3"
 date="2013-03-16T20:48:42Z"
 content="""
I don't know if it will help, but XA support was turned off in hope that it would alleviate the random presence message loss I keep seeing with Google Talk, and since it didn't seem to help solve that problem, I've turned it back on.
"""]]


code:
### Please describe the problem.
Tried adding a removable drive repository through git-annex assistant on Mac, asked to encrypt it, got Internal server error with explanation: unable to determine gcrypt-id of remote

### What steps will reproduce the problem?
current (today's) version of git-annex build for Mountain Lion installed
app launched
default annex initialized, a couple file added
a removable 2GB USB drive (ExFAT) mounted in OS
click Add another repository
choose removable drive
choose to encrypt
a window explaining the need to wait for entropy shows
then we get the Internal server error: unable to determine gcrypt-id of remote

### What version of git-annex are you using? On what operating system?
git-annex version 4.20130922-g7dc188a
Mac OS X 10.8.5

### Please provide any additional information below.

looking at transcript below, it appears the root cause is the lack of initial git/gcrypt configuration. perhaps this needs to be caught/addressed somehow

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

gpg: can't open `/usr/local/share/gnupg/options.skel': No such file or directory
.......+++++
.+++++
gpg: key 71BFBC31 marked as ultimately trusted
ok
(Recording state in git...)
(Recording state in git...)
(encryption setup) (hybrid cipher with gpg key C4B2EA8D71BFBC31) gcrypt: Development version -- Repository format MAY CHANGE
gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
gcrypt: WARNING: Skipping missing key C4B2EA8D71BFBC31
gcrypt: You have not configured any keys you can encrypt to for this repository
gcrypt: Use ::
gcrypt:   git config gcrypt.participants YOURKEYID
gcrypt: Development version -- Repository format MAY CHANGE
gcrypt: WARNING: Skipping missing key C4B2EA8D71BFBC31
gcrypt: You have not configured any keys you can encrypt to for this repository
gcrypt: Use ::
gcrypt:   git config gcrypt.participants YOURKEYID
03/Oct/2013:00:05:24 +0400 [Error#yesod-core] unable to determine gcrypt-id of remote @(yesod-core-1.1.8.3:Yesod.Internal.Core ./Yesod/Internal/Core.hs:550:5)

# End of transcript or log.
"""]]

[[!tag moreinfo]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 1"
 date="2013-10-02T20:37:03Z"
 content="""
Can you take a look at ~/annex/.git/config and if there is an entry for this git remote, post it? (Or post the whole thing if unsure..)

Also, can you get gpg to list your secret keys, and see if key 71BFBC31 is in the list? You can do so by running `gpg --list-secret-keys` ; if you don't have gpg in your path at the terminal, you can use the version from the git-annex DMG by first running `/Volumes/git-annex/git-annex.app/Contents/MacOS/runshell`
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnyMzZZLS1xGW1raqc_9Md6Ksdkvx5rUJU"
 nickname="Michael"
 subject="comment 2"
 date="2013-10-04T10:59:30Z"
 content="""
Sorry Joey, I was tinkering around trying to get the assistant off the ground and have destroyed the evidence of the bug in the process. I will try to recreate it and post the details. (most of my problems are/were related to bundled git/gpg/other software, I posted a separate comment in OSX Install page)
"""]]


code:
### Please describe the problem.
Setting up a git annex repo with the webapp. Adding a small archive removeable disk.
Adding a file to repo. Moving file to archive. The file seems to get copied to the removeable disk
but it is not removed (dropped)from my repo, as I would expect

Before I add the file to the repo, the archive directory on the removeable drive looks like a bare repo.
After adding one file and moving it to the archive folder, a ".git" directory gets created inside the bare repo.
The .git directory contains an annex directory, and nothing else.

Is this behaviour the expected one? As far as I have understood it, the files are supposed to "disapear" from my computer when I move them to the archive folder.

### What steps will reproduce the problem?
Se above

### What version of git-annex are you using? On what operating system?
Version: 5.20131127-g736ce5e
Build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus XMPP Feeds Quvi TDFA

Chakra linux
Linux black 3.10.18-1-CHAKRA #1 SMP PREEMPT Sat Nov 9 20:18:15 UTC 2013 x86_64 GNU/Linux

git annex installed from prebuilt tar.

### Please provide any additional information below.

[[!format sh """
[2013-11-28 23:25:59 CET] main: starting assistant version 5.20131127-g736ce5e
[2013-11-28 23:25:59 CET] Cronner: You should enable consistency checking to protect your data.
(scanning...) [2013-11-28 23:25:59 CET] Watcher: Performing startup scan
(started...) gcrypt: Development version -- Repository format MAY CHANGE
gcrypt: Repository not found: /run/media/daniel/7090f0c1-abf9-47a1-a186-ba243c498172/test_ark
fatal: Not a git repository: '/run/media/daniel/7090f0c1-abf9-47a1-a186-ba243c498172/test_ark/.git'
[2013-11-28 23:26:21 CET] main: Syncing with 7090f0c1abf947a1a186ba243c498172
warning: no common commits
From /run/media/daniel/7090f0c1-abf9-47a1-a186-ba243c498172/test_ark
 * [new branch]      git-annex  -> 7090f0c1abf947a1a186ba243c498172/git-annex
 * [new branch]      master     -> 7090f0c1abf947a1a186ba243c498172/master
Already up-to-date.
To /run/media/daniel/7090f0c1-abf9-47a1-a186-ba243c498172/test_ark
 * [new branch]      git-annex -> synced/git-annex
 * [new branch]      annex/direct/master -> synced/master
Already up-to-date.
Already up-to-date.
[2013-11-28 23:26:59 CET] Pusher: Syncing with test_ark
To /run/media/daniel/7090f0c1-abf9-47a1-a186-ba243c498172/test_ark
   4d435e3..e548577  git-annex -> synced/git-annex
[2013-11-28 23:27:17 CET] Committer: Adding git-annex..64.tar.gz

(Recording state in git...)
(merging 7090f0c1abf947a1a186ba243c498172/git-annex into git-annex...)
(Recording state in git...)

(Recording state in git...)
add git-annex-standalone-amd64.tar.gz (checksum...) [2013-11-28 23:27:17 CET] Committer: Committing changes to git
[2013-11-28 23:27:17 CET] Pusher: Syncing with test_ark
Already up-to-date.
To /run/media/daniel/7090f0c1-abf9-47a1-a186-ba243c498172/test_ark
   e548577..b86f41c  git-annex -> synced/git-annex
   658b0f2..f32a646  annex/direct/master -> synced/master
Already up-to-date.
[2013-11-28 23:27:27 CET] Committer: Adding git-annex..64.tar.gz
[2013-11-28 23:27:27 CET] Committer: Committing changes to git
[2013-11-28 23:27:27 CET] Pusher: Syncing with test_ark

git-annex-standalone-amd64.tar.gz

       32768   0%    0.00kB/s    0:00:00  Already up-to-date.
To /run/media/daniel/7090f0c1-abf9-47a1-a186-ba243c498172/test_ark
   b86f41c..806a2dd  git-annex -> synced/git-annex
   f32a646..5e254e7  annex/direct/master -> synced/master
Already up-to-date.

    20143752 100%   85.24MB/s    0:00:00 (xfer#1, to-check=0/1)

sent 20146306 bytes  received 31 bytes  40292674.00 bytes/sec
total size is 20143752  speedup is 1.00
fatal: Not a git repository: '/run/media/daniel/7090f0c1-abf9-47a1-a186-ba243c498172/test_ark/.git'
fatal: Not a git repository: '/run/media/daniel/7090f0c1-abf9-47a1-a186-ba243c498172/test_ark/.git'
fatal: Not a git repository: '/run/media/daniel/7090f0c1-abf9-47a1-a186-ba243c498172/test_ark/.git'
[2013-11-28 23:27:59 CET] Pusher: Syncing with test_ark
Everything up-to-date

# End of transcript or log.
"""]]

> [[fixed|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.64"
 subject="comment 1"
 date="2013-11-30T20:18:53Z"
 content="""
This bug, introduced in version 5.20131118, caused `git annex init`, when run in a bare repository, to set `core.bare` to false.

I have fixed the bug. I may need to make git-annex detect and automatically fix up after the problem.

To manually fix a bare repository created by the buggy version of git-annex, run `git config core.bare true` in the repository. Then if there's a .git/annex/objects inside the bare git repository, you can move it to annex/objects.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.64"
 subject="comment 2"
 date="2013-12-02T16:35:41Z"
 content="""
I've now made git-annex automatically detect these problem repositories, and fix them up. You should not need to do anything to recover, just upgrade git-annex eventually, and use it.
"""]]


code:
The assistant does not allow adding an existing repo: I tried to add a "remote server", to enter the credits, and then I add to choose between "git-annex" or "rsync". Choosing "git-annex" tries to create a new repo and then fails. The repo on the server is a bare one, accessed via gitolite. Setting the remote manually ("git remote add origin git@example.org:my-annex.git" works (I can pull, push, ...))

Mac OS 10.7 version 2013-09-10

[[!meta title="assistant does not interoperate with gitolite when adding a repository"]]

[2013-09-13 17:00:55 CEST] chat: ssh ["-p","22","git@example.org","sh -c 'mkdir -p '\"'\"'my-annex.git'\"'\"'&&cd '\"'\"'my-annex.git'\"'\"'&&if [ ! -d .git ]; then git init --bare --shared; fi&&git annex init'"]

[[!tag confirmed]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.2.134"
 subject="comment 1"
 date="2013-09-13T18:19:03Z"
 content="""
I suppose the assistant could try to add a git remote and pull from it, and only if this pull fails go try to run shell commands to create it. This might work with gitolite, gitosis, github, etc. Although only partially for ones that don't support running git-annex-shell..
"""]]


code:
git-annex has nice repository cost tracking, but the assistant doesn't
always use this when queuing downloads.

For example, if a network comes up, it'll queue a lot of downloads from a
remote, perhaps expensive server. If a removable drive is then attached,
which has a lower cost, it'll queue downloads from there, but
only once it works through all the ones it's already queued from the
network.

There is a workaround: Since the webapp allows disabling syncing to a
remote, the user can go in and disable syncing to the network remote
temporarily.

[[!tag /design/assistant]]

--[[Joey]]

[[!tag confirmed]]


code:
### Please describe the problem.

The assistant seems to only sync files and not directories, so empty directories will not be synced.

### What steps will reproduce the problem?

- Create an empty directory on HostA
- See that nothing gets synced (daemon.log is silent) and the directory doesn't exist on the other HostB
- Create an empty file inside the empty directory on HostA
- See that now HostB has the empty file inside the directory

I'm sure this comes directly from the git behavior but is at least surprising for a direct mode repository using the assistant.

### What version of git-annex are you using? On what operating system?

[[!format sh """
$ git annex version
git-annex version: 4.20130516.1
build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP
local repository version: 4
default repository version: 3
supported repository versions: 3 4
upgrade supported from repository versions: 0 1 2
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 12.04.2 LTS
Release:	12.04
Codename:	precise
"""]]

> [[!taglink confirmed]] (but may be out of scope) --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-05-31T21:14:11Z"
 content="""
Git does not have a concept of an empty directory. The  typical workaround is to put a .gitignore file in the directory.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkx5V3MTbzCXS3J7Mn9FEq8M9bPPYMkAHY"
 nickname="Pedro"
 subject="comment 2"
 date="2013-05-31T21:48:24Z"
 content="""
Adding .gitignore files makes sense for a git standpoint but whatever workaround is needed to make the assistant works properly should be done by the assistant itself no? Maybe having direct mode commit a shaddow \".this_is_a_magic_empty_file\" in empty directories and then not actually writing that file into direct mode repositories on the other side.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmOsy6nbvPyXLd--qqjPMLnVIzxgZwtKlQ"
 nickname="Nicolas"
 subject="comment 3"
 date="2013-09-11T23:19:26Z"
 content="""
This behaviour is indeed very confusing when one attempts to use git-annex with the assistant only, without thinking about the underlying machinery.
Having the assistant automatically put a .gitignore file in empty directories is a possible solution, but maybe printing a warning in the log when the watcher sees an empty directory could be an unobtrusive and helpful solution? (At least it would have helped me)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmOsy6nbvPyXLd--qqjPMLnVIzxgZwtKlQ"
 nickname="Nicolas"
 subject="comment 4"
 date="2013-09-11T23:24:58Z"
 content="""
Another problem with the current behaviour is that when deleting a directory and its contents on a repository, the contents are deleted on the other repositories but empty directories are left behind...
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmNu4V5fvpLlBhaCUfXXOB0MI5NXwh8SkU"
 nickname="Adam"
 subject="comment 5"
 date="2013-11-01T22:12:04Z"
 content="""
I agree that this is very confusing.  Also having to manually delete empty, renamed directories is...less than ideal.  :)
"""]]


code:
### Please describe the problem.

The assistant does not sync file permissions so if you set something as executable it won't be set the same on the other repositories.

### What steps will reproduce the problem?

On the first host:

[[!format sh """
$ echo -e '#!/bin/sh\necho "Hello World!"' > testscript
$ chmod ugo+x testscript
$ ./testscript
Hello World!
$ ls -l
total 4
-rwxr-xr-x 1 pedrocr pedrocr 30 May 31 20:00 testscript
"""]]

And on the second host:

[[!format sh """
$ ls -l
total 4
-rw-r--r-- 1 pedrocr pedrocr 30 May 31 20:00 testscript
$ ./testscript
bash: ./testscript: Permission denied
"""]]

### What version of git-annex are you using? On what operating system?

[[!format sh """
$ git annex version
git-annex version: 4.20130516.1
build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP
local repository version: 4
default repository version: 3
supported repository versions: 3 4
upgrade supported from repository versions: 0 1 2
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 12.04.2 LTS
Release:	12.04
Codename:	precise
"""]]

> [[!taglink confirmed]] (but may be out of scope) --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-05-31T21:17:17Z"
 content="""
More generally, symbolic links cannot have permissions of their own, and git in fact indicates a file is a symlink by setting the tree object's mode to a specific magic number. So there is no git metadata that git-annex can use to track whether a file is supposed to be executable.

Therefore, this will only get fixed when there's some kind of full-fledged metadata storage for git.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkx5V3MTbzCXS3J7Mn9FEq8M9bPPYMkAHY"
 nickname="Pedro"
 subject="comment 2"
 date="2013-05-31T21:45:39Z"
 content="""
Note that the file I created wasn't a symlink. I assume you mention symlinks because git-annex itself checks in symlinks to git instead of actual files?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmNu4V5fvpLlBhaCUfXXOB0MI5NXwh8SkU"
 nickname="Adam"
 subject="comment 3"
 date="2013-11-02T23:49:49Z"
 content="""
Dropbox handles permissions, e.g. if I \"chmod -x\" a file on one system, it does the same to my other systems.  It would be a bit of a step backward if git-annex can't do this.  :/  Just my two cents.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8"
 nickname="TobiasTheViking"
 subject="comment 4"
 date="2014-01-12T19:16:07Z"
 content="""
Yeah, now i just hit this. kinda annoying.

Couldn't this at least be solved for direct mode repositories?

"""]]


code:
[[!comment format=mdwn
 username="http://cstork.org/"
 nickname="Chris Stork"
 subject="comment 5"
 date="2014-01-14T08:33:21Z"
 content="""
@Pedro: What Joey is refering to is that git-annex uses sym-links behind the scenes to store even such a small file.  (BTW, that can be changed with the large-file config option, IIRC.)

@Joey: Even a \"full-fledged metadata storage for git\" wouldn't solve this problem since from git's POV it's still just a sym-link without permissions.
"""]]


code:
[[!comment format=mdwn
 username="Xyem"
 ip="178.79.137.64"
 subject="comment 6"
 date="2014-06-24T10:05:26Z"
 content="""
Now that git-annex has a proper metadata system now, can this be handled?

Perhaps doing this:

git annex metadata -s __executable 001

could cause git-annex to set the executable bit (on get, fsck etc.) and that be set automatically on git annex add if the file has executable bit set (numbers are a bitmask for which execute bit to set..?).
"""]]