code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 1"
 date="2013-12-24T20:51:30Z"
 content="""
How many files are in this repository?

There are various ways to profile git-annex's memory use, but they all require building it from source. For example, it can be built with [[EKG_support|ekg]].
"""]]

repo_name: kakafarm/git

license: unknown


code:
[[!comment format=mdwn
 username="https://launchpad.net/~psycojoker"
 nickname="psycojoker"
 subject="comment 2"
 date="2014-01-04T04:43:26Z"
 content="""
There are 139 files in this repo but this repos is updated very frequently (the files are generated every 10min).

After a lot of fight with cabal and cabal-dev, I've managed to make EKG run, here is a screenshotof the result after hours of run and an actual consumption of 33.8% of 8gb of ram.

http://pix.toile-libre.org/upload/original/1388810323.png

But looking at it I don't know how it can be usefull. I don't know very well the haskell ecosystem, don't hesitate to point me to things that could help.

On a more positive note: git annex compiled from cabal (and from source) seems to eat my available memory way slower than the one from the ubuntu ppa.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.35"
 subject="comment 3"
 date="2014-01-05T20:41:35Z"
 content="""
Well, EKG either cannot see the leak, or perhaps haskell profiling generally cannot see it. Don't have a lot of experience with EKG, it's just a lot easier to ask for than full ghc profiling data..

It's certainly useful information that you are constantly changing the files.

I have tried to replicate this setup, with a few hundred files, and writing random stuff to them repeatedly. The RSS did not grow at all despite repeatedly changing and committing the files.

I then tried adding a remote for it to sync with, and see evidence of a small leak there -- around 8 bytes per file transferred. The leak appears to be in the transfer code, not the git sync code; it happens with both git remotes and with directory special remotes.

"""]]


code:
[[!comment format=mdwn
 username="https://launchpad.net/~psycojoker"
 nickname="psycojoker"
 subject="comment 4"
 date="2014-01-06T01:31:14Z"
 content="""
Great, don't hesitate to ask me if you need more informations!

Thanks :)
"""]]


code:
### Please describe the problem.

`git annex status` inconsistently uses mebi (SI) and giga (informal) prefixes.

### What steps will reproduce the problem?

Example:

[[!format txt """
anarcat@marcos:mp3$ git annex status
supported backends: SHA256E SHA1E SHA512E SHA224E SHA384E SHA256 SHA1 SHA512 SHA224 SHA384 WORM URL
supported remote types: git S3 bup directory rsync web glacier hook
repository mode: direct
trusted repositories: 0
semitrusted repositories: 2
        00000000-0000-0000-0000-000000000001 -- web
        b7802161-c984-4c9f-8d05-787a29c41cfe -- here (anarcat@marcos:/srv/mp3)
untrusted repositories: 0
transfers in progress: none
available local disk space: 31.93 gigabytes (+1 megabyte reserved)
local annex keys: 19913
local annex size: 111.08 gigabytes
known annex keys: 20085
known annex size: 111.38 gigabytes
bloom filter size: 16 mebibytes (3.1% full)
backend usage:
        SHA256E: 39998
"""]]

Notice `mebibytes` and `gigabytes`. It is unclear whether those are gigabytes (1000^3) or gibibytes (1024^3).

### What version of git-annex are you using? On what operating system?

4.20130802~bpo70+2, Debian wheezy.

### Please provide any additional information below.

nil. --[[anarcat]]

> git-annex consistently uses the powers of ten units
> for disk storage sizes. Its "gigabyte" is the SI gigabyte.
>
> It uses the absurdly named units for powers of 2 for memory sizes,
> in the few places it deals with memory (probably only the above bloom
> filter size number).
>
> AFAIK I am complying with all relevant standards and best practices.
> Even though I consider them rather dumb, as is clear if you
> [read the opionated source code I wrote to handle this]().
>
> If git-annex used "gibibyte", the numbers it reports for disk size
> would not match the numbers disk vendors and most tools use.
>
> [[bug_is_in_world_not_in_git-annex|done]] --[[Joey]]


code:
### Please describe the problem.

There are multiple problems that have the same cause:

 * When using a shared repository (core.sharedrepository = group), the directories that contain the actual objects may be owned by different users. In this case adding or removing the sticky bit is prohibited by the operating system. Thus shared repositories can only be incrementally fscked if all objects are owned by the same user.

 * It is not currently possible to run incremental fscks on the local repository and on a special remote at the same time, because both of them use the same flag space, the sticky bit.

### What steps will reproduce the problem?

Create a shared repository (core.sharedrepository = group), let a different user add an object and then try to fsck it.

### What version of git-annex are you using? On what operating system?

Debian's 4.20131106~bpo70+1

[[!tag confirmed]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.64"
 subject="comment 1"
 date="2013-12-02T17:58:55Z"
 content="""
See [[todo/incremental_fsck]] for background.

Options:

* using per-object files
* using sqllite or another relational database
* using git as the database
* using a key/value store

Per-object files have the problem that they clutter up the .git/annex/objects/ tree quite a lot with info only used by fsck. We have a similar clutter problem with direct mode mapping files, and for various reasons (including [[direct mode mappings scale badly with thousands of identical files|bugs/\"Adding_4923_files\"_is_really_slow]]) I have been hoping to move that data to a better storage, perhaps a database, eventually. Switching fsck to use a database first might be a good first step to using a database for the more important direct mode mapping storage. If the fsck database goes wrong, the worse that happens is some extra incremental fscking.

Using git as the database is possible. Just store the info in a separate git ref, like the git-annex branch, but that is not synced. But it will use more disk over time. Probably not the best choice.

There are plenty of sqllite interfaces for haskell. They all do have the problem that the C sqlite has to be installed. This will make `cabal install git-annex` harder. It is currently possible to do a `cabal install git-annex` with flags that avoid needing to install any C libraries. This is useful for my sanity, since otherwise people want hand-holding on installing libraries on OSX and stuff.

Perhaps there's a pure haskell key/value store that would be a better choice than sqllite. I do not anticipate git-annex needing complex relational database storage, if we look at everything it needs to store so far, key/value is enough. (The more complicated stuff is stored in git anyway.)

* <http://hackage.haskell.org/package/io-storage> is actually memory only, not suitable
* <http://hackage.haskell.org/package/acid-state> is I think the gold standard in its area. Strongly typed. Has some Template Haskell stuff but it's optional. Packaged in Debian already, but only on TH capable architectures so that would need to be fixed. Seems to have the disadvantage that it's not really a key/value store, so works by loading the *entire* data into ram. This is a problem since git-annex wants to run in constant memory.
* <http://hackage.haskell.org/package/keyvaluehash> has some caveats about not being able to remove old keys
* <http://hackage.haskell.org/package/HongoDB> looks possible, has only had one release in 2011 though and undocumented
"""]]


code:
[[!comment format=mdwn
 username="helmut"
 ip="89.0.78.89"
 subject="git as a database"
 date="2013-12-02T18:24:12Z"
 content="""
You said that \"[...] it will use more disk over time. Probably not the best choice.\" But this does not have to be true. Since this git ref is never synced, an update can be forced. The tricky part is in finding good truncation points. Now for fsck, the log can be truncated whenever an incremental fsck is started.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.64"
 subject="comment 3"
 date="2013-12-02T19:11:27Z"
 content="""
HongoDB doesn't build with current hackage.

keyvaluehash passed some inital load etc tests. The inability to delete old keys/values might not be a significant problem.

However I don't think it has any ACID guarantees; eg a crash at the wrong time could leave the database in some inconsistent state. <https://github.com/Peaker/keyvaluehash/issues/1>
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.64"
 subject="comment 4"
 date="2013-12-02T20:35:45Z"
 content="""
@helmut Well, true, it could overwrite the old ref with a new one at the start of a new fsck pass. If there was a separate ref for each uuid, this would work even for mixed local/remote fscking. And git should eventually drop the objects associated with the old ref, probably in `gc --auto`, which has a default --prune of 2 weeks.

The git overhead would probably be fine for fsck. I am less sure about it for the direct mode mapping files, which can be queried quite frequently. In particular inAnnex is used lots of places, often repeatedly on a lot of files, and it uses withObjectLoc, which uses associatedFiles.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="68.119.96.78"
 subject="comment 5"
 date="2013-12-21T16:26:51Z"
 content="""
<http://hackage.haskell.org/package/TCache> looks potentially good. It seems that it may allow storing stuff on disk and only loading it into cache when accessed, but I am not sure. Need to run some tests to try it out.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.172"
 subject="comment 6"
 date="2014-02-20T20:09:22Z"
 content="""
I also need some local fast database for metadata index storing.
"""]]


code:
### Please describe the problem.

I get an error in one of my repositories.

### What steps will reproduce the problem?

I paired my macbook repository with an repository on my desktop. The repository on the desktop contains many smaller files, because it is my documents folder.

Pairing this repository with smaller files never worked for me. I posted to the forum about it:
[[forum/ssh__95__exchange__95__identification:_read:_Connection_reset_by_peer/]]

I am a little confused because I paired three other repositories between these computers sucessfully. They contain pictures, videos and music.

### What version of git-annex are you using? On what operating system?

Version: 5.20140106-gcb3351b
Build flags: Assistant Webapp Pairing Testsuite S3 WebDAV FsEvents XMPP DNS Feeds Quvi TDFA CryptoHash

Mac OS X Mavericks

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

Please make sure you have the correct access rights
and the repository exists.
fatal: index file smaller than expected
TransferScanner crashed: user error (git ["--git-dir=/Users/juh/Share/.git","--work-tree=/Users/juh/Share","-c","core.bare=false","update-index","-z","--index-info"] exited 128)
[2014-01-10 16:26:00 CET] TransferScanner: warning TransferScanner crashed: user error (git ["--git-dir=/Users/juh/Share/.git","--work-tree=/Users/juh/Share","-c","core.bare=false","update-index","-z","--index-info"] exited 128)
ssh_exchange_identification: read: Connection reset by peer
ssh_exchange_identification: read: Connection reset by peer
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
[2014-01-10 16:56:09 CET] NetWatcherFallback: Syncing with sokrates.local_Dokumente
ssh_exchange_identification: read: Connection reset by peer
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: index file smaller than expected
NetWatcherFallback crashed: user error (git ["--git-dir=/Users/juh/Share/.git","--work-tree=/Users/juh/Share","-c","core.bare=false","update-index","-z","--index-info"] exited 128)
[2014-01-10 17:01:09 CET] NetWatcherFallback: warning NetWatcherFallback crashed: user error (git ["--git-dir=/Users/juh/Share/.git","--work-tree=/Users/juh/Share","-c","core.bare=false","update-index","-z","--index-info"] exited 128)

# End of transcript or log.
"""]]

> Made assistant detect corrupted .git/annex/index on startup and correct
> it. [[fixed|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.43"
 subject="comment 1"
 date="2014-01-14T21:12:19Z"
 content="""
It's unusual for git's index file to get corrupted or short like this. git writes to .git/index by first writing the new content to .git/index.lock, and then once it's written, renaming it. So if git is interrupted in the middle of a write, it doesn't leave the index file truncated. Of course, it's somewhat up to the OS's filesystem and buffering, and so I suppose if the system loses power at just the right time, and the filesystem does not journal data, this could happen.

Anyway, git-annex's assistant should be able to detect when the index file is corrupt, including too short, and fix it. When I try with a current version of git-annex, opening the webapp or starting the assistant results in the index file being automatically repaired, with this logged to .git/annex/daemon.log:

<pre>
fatal: index file smaller than expected
[2014-01-14 16:58:11 JEST] SanityCheckerStartup: corrupt index file found at startup; removing and restaging
</pre>

However, it looks like it needs to do the same check for .git/annex/index. I was able to reproduce this bug by corrupting that file.
"""]]


code:
### Please describe the problem.

The script producing the .info file along with the binary tarball seems to generate the wrong version.

### What steps will reproduce the problem?

    % curl https://downloads.kitenet.net/git-annex/linux/current/git-annex-standalone-amd64.tar.gz.info
    GitAnnexDistribution {distributionUrl = "https://downloads.kitenet.net/git-annex/linux/current/git-annex-standalone-amd64.tar.gz",
    distributionKey = Key {keyName = "2a01f853b08dc0f47392b50cda41d5f589f46413073fdf4ffd5b817d6638a502.tar.gz", keyBackendName = "SHA256E",
    keySize = Just 27578790, keyMtime = Nothing}, distributionVersion = "5.20131222",
    distributionReleasedate = 2013-12-30 18:56:44.908259 UTC, distributionUrgentUpgrade = Nothing}

`distributionVersion` is 5.20131222, but `git annex version` returns 5.20131230-g4aa88d8.

This may look anecdotical, but at least the `git-annex-bin` package of Arch Linux uses the .info data to retrieve the version information.

> [[fixed|done]] --[[Joey]]


code:
### Please describe the problem.

When working with git-annex in a folder managed by Dropbox, the folders names in .git/annex/objects/ are mangled by Dropbox. This leads to lost files.

This is not a git-annex bug, but the problem is severe and it would be helpful when git-annex can deal with it somehow. Possibilities are:

* include a warning in documentation (this will be probably found after someone encounters the problem actually)
* issue a warning when running in a Dropbox managed folder
* avoid using filenames that differs in case only (this would make git-annex also more robust against corruption i.e. when a repository is copied over filesystems/protocols without filename case support)

Dropbox file name limitations: [[https://www.dropbox.com/help/145/en]]

### What steps will reproduce the problem?
    cd ~/Dropbox
    mkdir test_annex
    cd test_annex
    git init
    git annex init "test_annex"
    cp ~/somefiles/* .
    git annex add *
    # wait for Dropbox to upload and mangle folder names in .git/annex/objects/
    find -L . -type l | wc -l # print number of broken symlinks
    git annex fsck # prints all the missing files

### What version of git-annex are you using? On what operating system?

* git-annex 3.20130207
* Fedora 19

### Please provide any additional information below.

[[!format sh """
# folder names mangled by Dropbox

$ ls .git/annex/objects/
Ff/                      M9 (Case Conflict (1))/  x8 (Case Conflict (1))/
Ff (Case Conflict)/      pF/                      x9/
Ff (Case Conflict (1))/  pF (Case Conflict)/      x9 (Case Conflict)/
Ff (Case Conflict (2))/  pF (Case Conflict (1))/  Zf/
Ff (Case Conflict (3))/  pF (Case Conflict (2))/  Zf (Case Conflict)/
Ff (Case Conflict (4))/  pF (Case Conflict (3))/  Zf (Case Conflict (1))/
Ff (Case Conflict (5))/  pF (Case Conflict (4))/  Zf (Case Conflict (2))/
Ff (Case Conflict (6))/  PG/                      Zf (Case Conflict (3))/
...

# End of transcript or log.
"""]]

> [[done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.35"
 subject="comment 1"
 date="2014-01-06T16:02:45Z"
 content="""
I'm not seeing data loss. It looks like the contents of your files are still there, you just need to rename them back to the right directory names. If you're using indirect mode, you should be able to do this:

    mv .git/annex/objects .
    chmod -R u+w objects
    git annex add objects

That will re-inject all the file contents with the right names. You can then delete the objects directory, which should only contain symlinks at that point.

----

Moving git-annex to using lower case hash directories is on the very long term todo list, due to various crappy filesystems (problems mostly worked around by direct mode). Since it requires changing every symlink in every existing git-annex repository, which will be extremely disruptive, I'd need a much better reason than dropbox to do it now.

I have no interest, in general, in making git-annex use filenames that meet whatever limitations dropbox wants to impose on its users.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkWQXo7MLYIDa-ep04UTbdz8KuvfjwLDCQ"
 nickname="Matěj"
 subject="comment 2"
 date="2014-01-08T22:11:34Z"
 content="""
I understand, it's not git-annex fail. I added at least a tip with a warning here: [[forum/do not use git-annex inside your Dropbox]].
"""]]


code:
### Please describe the problem.

Some logs fail to be displayed, and instead of displaying parts of the log, no logs at all are displayed in the webapp.

The problem character is, I believe, a latin-1 encoded filename (as opposed to UTF-8). --[[anarcat]]

### What steps will reproduce the problem?

1. download [this logfile](http://paste.anarc.at/daemon.log.1)
2. install it in .git/annex/daemon.log
3. load the webapp
4. visit the logs page

### What version of git-annex are you using? On what operating system?

4.20131105-g8efdc1a

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
Internal Server Error
/srv/video/.git/annex/daemon.log.1: hGetContents: invalid argument (invalid byte sequence)
git-annex version 4.20131105-g8efdc1a
# End of transcript or log.
"""]]

> [[fixed|done]] --[[Joey]]


code:
### Please describe the problem.
Some internals of git annex does not check if the shell it is running is Posix-compatible, ie. bash.

I am using fish, and after setting up local pairing, and working, I switched back the login-shell to fish, and when syncing a file, I got this error, read from daemon.log:

fish: Unknown command 'GIT_ANNEX_SHELL_DIRECTORY=/home/s/annex'. Did you mean to run ~/.ssh/git-annex-shell with a modified environment? Try 'env GIT_ANNEX_SHELL_DIRECTORY=/home/s/annex ~/.ssh/git-annex-shell...'. See the help section on the set command by typing 'help set'.
Standard input: GIT_ANNEX_SHELL_DIRECTORY='/home/s/annex' ~/.ssh/git-annex-shell
                ^
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

### What steps will reproduce the problem?
Set up local pairing ( I believe having sh/bash as login terminal is necessary for this).
Switch back to fish as login-shell with chsh -s /usr/bin/fish
Add a file to either repository.

### What version of git-annex are you using? On what operating system?
[s@b ~]$ git annex version
git-annex version: 5.20140320-g63535e3
build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV Inotify DBus XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier hook external

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

# End of transcript or log.
"""]]

> [[fixed|done]] so
>
> I have not tried to make the assistant go back and fix up existing
> `authorized_keys` lines. So if someone had been using a posix shell and
> switched to fish, they'll hit this and need to fix it themselves. I judge
> this is pretty small number of users. --[[Joey]]


code:
[[!comment format=mdwn
 username="http://johan.kiviniemi.name/"
 nickname="Johan"
 subject="comment 1"
 date="2014-04-02T08:40:17Z"
 content="""
FWIW, `env foo=bar cmd` would probably work on every system on which the `foo=bar cmd` invocation works now.
"""]]


code:
### Please describe the problem.

I noticed that you (@joey) wrote

> webapp: Odd problem when using Firefox: Some links don't seem to load. Clicking on such a link repeatedly will eventually load the right page. Network console shows web browser is requesting page "/" despite a link to another page being clicked on. Workaround: Use Chrome instead; no such problem there.

in [[todo/windows_support]], and thought I would get around to mentioning that I also see this problem on linux (so it's not windows specific), but I am using firefox.

### What steps will reproduce the problem?
Click on a link in the webapp; most of the time, even though the link target is correct, the current page is reloaded instead of loading the target.

### What version of git-annex are you using? On what operating system?
5.20131205-gc448602

Ubuntu 13.04, firefox 25.0.1

--Walter

> [[fixed|done]]; increased number of longpolling failures allowed before
> it enters the chromium back button bugfix workaround to 12.
> Should be more than enough for 3 or 4 long polling elements on a page.
> (Considered only running the chromium bugfix on chromium, but I don't
> want to get into browser detection hacks.) --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.64"
 subject="comment 1"
 date="2013-12-09T21:08:29Z"
 content="""
Thanks, that's very interesting indeed.

When I was debugging this on Windows, I had the network console open, and could see clearly that Firefox was trying to load \"/\" when I clicked on a link to \"/config\". Both firefox and the git-annex webapp's debug log agreed that it asked for the wrong page.

I have just reproduced it using iceweasel as well. I know that I have tested git-annex before under iceweasel and it seemed to work.

Seems that clicking 3+ times on a link makes it finally follow it.

I wonder if this has something to do with the long polling. Disabling javascript would avoid it if so. They've made disabling JS in firefox sufficiently hard that I cannot find how to do it.

Also, the problem seems to open happen on the main page (/). Once I get to any other page, further links are followed without trouble.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.64"
 subject="comment 2"
 date="2013-12-09T21:39:16Z"
 content="""
Tried renaming / to /dashboard , just in case firefox was having problems with the name of the page. Didn't help, so the problem must be something on the page.

Tried removing the repolist and the transferlist from the dashboard, and this indeed avoided the problem. So it's certainly related to the long polling code. Although note that this left 1 long polling call still on the page, for the sidebar. In fact, only removing one of the repolist or transfer list seems to be enough to avoid the problem. 1 works, 2 works, 3 fails.

Aha.. After 3 long polling failures, the JS runs a fail function. To work around a seeming bug in chromium when navigating back to a page, [[!commit b5678d74a29023cbeb3d65c0d7f57d20f47ebef9]] made this reload the page.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnSenxKyE_2Z6Wb-EBMO8FciyRywjx1ZiQ"
 nickname="Walter"
 subject="comment 3"
 date="2013-12-09T21:43:58Z"
 content="""
I just tried in the browser console, and see the same behaviour.

I also see that links work when not on the dashboard.

I disabled javasript (set javascript.enabled to false in about:config), and the links all work well (though dropdown menus don't work at all).

(I see after writing this you seem to have worked it out already)
"""]]


code:
What steps will reproduce the problem?
In the git-annex tree,

    $ make
    $ sudo make install

What is the expected output? What do you see instead?

One would expect git-annex to be installed in the configured prefix. Unfortunately, make tries to rebuild the "all" target before "install". This fails (presumably) due to incorrect environment state,

    ghc -O2 -Wall -ignore-package monads-fd -ignore-package monads-tf -outputdir tmp -IUtility -DWITH_ASSISTANT -DWITH_S3 -DWITH_INOTIFY --make git-annex Utility/libdiskfree.o

    Utility/FileSystemEncoding.hs:17:8:
        Could not find module `Data.Bits.Utils':
          Use -v to see a list of the files searched for.
    make: *** [git-annex] Error 1

Removing "all" from the dependencies of "install" allows the process to run to completion, although this is clearly not a satisfactory solution.

> [[fixed|done]] using an ugly stamp file. --[[Joey]]


code:
### Please describe the problem.
There is no git-annex-shell command in path and thus sync with ssh server fails. If I symlink /usr/bin/git-annex-shell -> /usr/bin/git-annex, it complains that key is already present in annex and errors out. There is executable git-annex-shell in server's ~/.ssh/

### What steps will reproduce the problem?
Start webapp to sync local repo with ssh server.

### What version of git-annex are you using? On what operating system?
git-annex 4.20130627 on Gentoo

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

[2013-07-08 17:28:48 EEST] main: starting assistant version 4.20130627
[2013-07-08 17:28:49 EEST] TransferScanner: Syncing with netbook_Lit, sigma_Lit

  dbus failed; falling back to mtab polling (SocketError {socketErrorMessage = "connect: does not exist (Connection refused)", socketErrorFatal = True, socketErrorAddress = Just (Address "unix:abstract=/tmp/dbus-HBxh6EyMJ3,guid=adc3101676daede2a128013351daa535")})
Already up-to-date.

(scanning...) [2013-07-08 17:28:49 EEST] Watcher: Performing startup scan
Already up-to-date.
Already up-to-date.

(started...) Everything up-to-date
Everything up-to-date
bash: git-annex-shell: command not found
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: remote command not found (code 127) at io.c(605) [sender=3.0.9]
bash: git-annex-shell: command not found
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: remote command not found (code 127) at io.c(605) [sender=3.0.9]

[2013-07-08 17:50:23 EEST] main: starting assistant version 4.20130627
[2013-07-08 17:50:23 EEST] TransferScanner: Syncing with netbook_Lit, sigma_Lit

  dbus failed; falling back to mtab polling (SocketError {socketErrorMessage = "connect: does not exist (Connection refused)", socketErrorFatal = True, socketErrorAddress = Just (Address "unix:abstract=/tmp/dbus-HBxh6EyMJ3,guid=adc3101676daede2a128013351daa535")})
Already up-to-date.

(scanning...) [2013-07-08 17:50:23 EEST] Watcher: Performing startup scan
Already up-to-date.
Already up-to-date.

(started...) Everything up-to-date
Everything up-to-date
git-annex-shell: key is already present in annex
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]
git-annex-shell: key is already present in annex
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]

# End of transcript or log.
"""]]

[[!tag done]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.254.222"
 subject="comment 1"
 date="2013-07-08T16:14:37Z"
 content="""
The Makefile contains, for the install target:

[[!format sh \"\"\"
        ln -sf git-annex $(DESTDIR)$(PREFIX)/bin/git-annex-shell
\"\"\"]]

How was git-annex installed on your server?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm01ida6POv7vqyUYtOlymEbJTbrImAIzM"
 nickname="Reinis"
 subject="comment 2"
 date="2013-07-08T18:09:50Z"
 content="""
That is Gentoo box and git-annex is installed using the following ebuild:

[[https://github.com/gentoo-haskell/gentoo-haskell/blob/master/dev-vcs/git-annex/git-annex-4.20130627.ebuild]]

which uses cabal to install it. It uses also the following eclass:

[[http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/eclass/haskell-cabal.eclass?revision=1.40&view=markup]]

The main functions of interest there are cabal-bootstrap(), cabal-copy() and cabal_src_install(). I'm not familiar enough with haskell nor with cabal to follow trough in detail how the setup program is constructed and how it works, but the end result is that this is all that is installed. I don't know if normal cabal installs work as intended, maybe something there needs to be adjusted?

[[!format sh \"\"\"
 * Contents of dev-vcs/git-annex-4.20130627:
/usr
/usr/bin
/usr/bin/git-annex
/usr/lib64
/usr/lib64/ghc-7.6.3
/usr/lib64/ghc-7.6.3/gentoo
/usr/lib64/ghc-7.6.3/gentoo/git-annex-4.20130627.conf
/usr/share
/usr/share/doc
/usr/share/doc/git-annex-4.20130627
/usr/share/doc/git-annex-4.20130627/COPYRIGHT.bz2
\"\"\" ]]
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.7.235"
 subject="comment 3"
 date="2013-07-08T18:22:18Z"
 content="""
`cabal install` also creates the git-annex-shell symlink. I have filed a bug on the ebuild: <https://github.com/gentoo-haskell/gentoo-haskell/issues/247>
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm01ida6POv7vqyUYtOlymEbJTbrImAIzM"
 nickname="Reinis"
 subject="comment 4"
 date="2013-07-08T18:44:59Z"
 content="""
Thanks! That leaves the issue that even when manually creating the symlink it still errors out (second half of the log in original report), but that probably is another issue. Should I open separate report for it? There was something similar in [[http://git-annex.branchable.com/bugs/Rsync_remote_created_via_webapp_remains_empty/]], but the command= part was in the file on my local computer only and removing it didn't make any difference.
"""]]


code:
### Please describe the problem.

The "import" section of the manpage has an indentation error.

### What steps will reproduce the problem?

man git-annex

### What version of git-annex are you using? On what operating system?

anything after 20130827, i believe.

### Please provide any additional information below.

silly patch:

[[!format diff """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
diff --git a/Build/mdwn2man b/Build/mdwn2man
index ba5919b..b3ee404 100755
--- a/Build/mdwn2man
+++ b/Build/mdwn2man
@@ -25,7 +25,7 @@ while (<>) {
                $inlist=1;
                $spippara=0;
        }
-       elsif (/.SH/) {
+       elsif (/^.SH/) {
                $skippara=0;
                $inlist=0;
        }
# End of transcript or log.
"""]]

-- [[anarcat]]

> [[applied|done]] --[[Joey]]


code:
### Please describe the problem.

`git annex merge` goes out of memory on a large repo.

### What steps will reproduce the problem?

    $ git annex merge
    merge git-annex git-annex: out of memory (requested 1048576 bytes)

### What version of git-annex are you using? On what operating system?

    git-annex version: 4.20130827
    build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP DNS

On debian wheezy i386

### Please provide any additional information below.

The repository contains a lot of files:

    $ cd .git/objects
    $ find . -type f | wc -l
    1091548

[[!tag moreinfo]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnxlx1UrzVhdy6_gFjzmF42x6QXxBUxg00"
 nickname="Jakukyo"
 subject="pre-commit hook of git annex"
 date="2013-09-11T00:58:09Z"
 content="""
The pre-commit hook of git annex throws an 'out of memory' error, too.

As a workaround, I've deleted ~/.git/hooks/pre-commit, and use `git merge` manually.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.2.134"
 subject="comment 2"
 date="2013-09-13T18:38:11Z"
 content="""
Can you run this so I can get a better idea of the size of this repository:

git ls-tree -r git-annex | wc -l
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.14.105"
 subject="comment 3"
 date="2013-09-19T18:53:14Z"
 content="""
I just fixed a memory leak that occurred when large files were checked directly into git (perhaps by accident by committing files manually when using direct mode). However, I don't think it affected `git annex merge`.

Still need information about how to reproduce this bug...
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnxlx1UrzVhdy6_gFjzmF42x6QXxBUxg00"
 nickname="Jakukyo"
 subject="size of repo"
 date="2013-09-21T09:32:03Z"
 content="""
    $ git ls-tree -r git-annex | wc -l
    29273
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.220"
 subject="comment 5"
 date="2013-09-25T18:36:39Z"
 content="""
That doesn't look very big, I merge one 3x that large on a 128 mb machine.

I think you will need to either email me privately so I can get a copy of your repository to investigate with ... or you can try to investigate on your own.

I think the first things I would try to debug this are to look over `git annex merge --debug` and see if I see anything unusual, and then I would probably `git checkout git-annex` in the repository, and wc -l on all the files and see if any file has a lot of lines, or is otherwise very large.

If that found nothing, my next step would be to rebuild git-annex from source with memory profiling enabled, as explained in this book, and try to get a memory profiling graph that explained what was using up the memory. <http://book.realworldhaskell.org/read/profiling-and-optimization.html>
"""]]


code:
Word summary: After migrating from SHA256 to SHA256E I still have a ton of SHA256 files around that aren't being found by unused.

Command outputs (see, specifically, the output of status showing number of SHA256 and SHA256E files):

    greg@eeepc:/mnt/blackbox/Media/Pictures/Photos$ less .gitattributes
    * annex.backend=SHA256E

    greg@eeepc:/mnt/blackbox/Media/Pictures/Photos$ git-annex migrate .
    (Recording state in git...)

    greg@eeepc:/mnt/blackbox/Media/Pictures/Photos$ git-annex unused
    unused . (checking for unused data...) (checking master...) (checking rose/master...) (checking x200s/master...)
      Some partially transferred data exists in temporary files:
        NUMBER  KEY
        1       SHA256E-s15766010--8132a02a8b245eb9842e89c5e696df4e9c82d676f8dec3c6bb96892c19f99d51.jpg

      To remove unwanted data: git-annex dropunused NUMBER

    ok
    (Recording state in git...)

    greg@eeepc:/mnt/blackbox/Media/Pictures/Photos$ git-annex status
    supported backends: SHA256E SHA1E SHA512E SHA224E SHA384E SHA256 SHA1 SHA512 SHA224 SHA384 WORM URL
    supported remote types: git S3 bup directory rsync web hook
    trusted repositories: 1
            c0e4106e-2631-11e2-9749-1bfa37a61069 -- rose
    semitrusted repositories: 3
            00000000-0000-0000-0000-000000000001 -- web
            9bd4077e-196c-11e2-9cc9-9faafb3e34ee -- x200s
            c69d6fcc-18d1-11e2-9487-2fe6dbf0516b -- here (photos on eeepc)
    untrusted repositories: 0
    dead repositories: 1
            3ebd5ac2-2092-11e2-856a-bb0203cce179 -- Photos on Rose
    transfers in progress:
            downloading 2011/06/30/IMG_8180.jpg
     from rose
    available local disk space: 2 terabytes (+1 megabyte reserved)
    temporary directory size: 9 megabytes (clean up with git-annex unused)
    local annex keys: 36210
    local annex size: 136 gigabytes
    known annex keys: 23388
    known annex size: 102 gigabytes
    bloom filter size: 16 mebibytes (7.2% full)
    backend usage:
            SHA256E: 37453
            SHA256: 22145
    (Recording state in git...)

    greg@eeepc:/mnt/blackbox/Media/Pictures/Photos$ apt-cache policy git-annex
    git-annex:
      Installed: 3.20121017
      Candidate: 3.20121017
      Version table:
     *** 3.20121017 0
            600 http://ftp.us.debian.org/debian/ unstable/main i386 Packages
            100 /var/lib/dpkg/status
         3.20120629 0
            650 http://ftp.us.debian.org/debian/ wheezy/main i386 Packages

As Joey predicted, this took care of itself over time. Marking as [[bugs/done]].


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.27"
 subject="comment 1"
 date="2012-11-07T19:02:45Z"
 content="""
Notice where it says:

> (checking master...) (checking rose/master...) (checking x200s/master...)

AFAICS, every one of those branches still refers to the migrated keys. For one thing, `git annex migrate` makes changes, but it doesn't commit them to master. And then you need to wait until the other remote are updated to.

Also, there's not really any point in worrying about getting rid of migrated keys, since `git-annex migrate` hard links the new key to the old key, so the old ones don't take up any additional disk space.

In other words: This should take care of itself as you continue to use the repository.
"""]]


code:
[[!comment format=mdwn
 username="http://grossmeier.net/"
 nickname="greg"
 subject="comment 2"
 date="2012-11-15T04:11:55Z"
 content="""
You were right, this has taken care of itself.

Thanks much for the solid piece of software.
"""]]


code:
This tags is for bugs needing more information from their submitter.
Please respond to the bug and provide the requested information.


code:
### Please describe the problem.
When I try to start the webapp, it fails, complaining about a nautilus script.

### What version of git-annex are you using? On what operating system?
Mythbuntu 12.04 (which is based on XFCE and doesn't have nautilus)
$ git-annex version
git-annex version: 5.20140402
build flags: Assistant Webapp Webapp-secure Pairing S3 WebDAV Inotify DBus DesktopNotify XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier hook external
local repository version: 5
supported repository version: 5
upgrade supported from repository versions: 0 1 2 4

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
$ git-annex webapp

git-annex: /home/mythbuntu/.local/share/nautilus/scripts/git-annex get: openFile: does not exist (No such file or directory)
failed
git-annex: webapp: 1 failed

# End of transcript or log.
"""]]

[[fixed|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmqz6wCn-Q1vzrsHGvEJHOt_T5ZESilxhc"
 nickname="Sören"
 subject="comment 1"
 date="2014-04-06T11:34:37Z"
 content="""
The problem has been reported [here](http://git-annex.branchable.com/bugs/git-annex_fails_to_start_when_nautilus_script_directory_is_missing/) as well and is already fixed in the latest [release](http://git-annex.branchable.com/news/version_5.20140405/).

"""]]


code:
### Please describe the problem.

git annex installer on Windows only installs git-annex-licenses.txt  git-annex-uninstall.exe  git-annex.exe

This makes git-annex unusable over ssh (Windows box runs cygwin sshd).

### What steps will reproduce the problem?

### What version of git-annex are you using? On what operating system?

Windows 7 64bit

git annex version
git-annex version: 5.20140421-g78d6aa1
build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier hook external

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

# End of transcript or log.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 1"
 date="2014-05-16T18:12:34Z"
 content="""
You should be able to copy git-annex.exe to git-annex-shell.exe and that will be usable as git-annex-shell. I could make the windows installer do this, but without the ability to easily hard or symlink the files, it would double the disk space for what seems an unusual use case..
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="comment 2"
 date="2014-05-16T18:58:13Z"
 content="""
Hmm, I don't understand the double disk space issue.
If direct mode is used, files are stored once, right?

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 3"
 date="2014-05-16T20:36:02Z"
 content="""
I mean that copying git-annex.exe to git-annex-shell.exe doubles the disk space needed to install git-annex.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="comment 4"
 date="2014-05-24T21:26:24Z"
 content="""
I tried copying git-annex.exe to git-annex-shell.exe, but it doesn't behave as git-annex-shell:

git-annex-shell -c configlist
git-annex-shell.exe: unknown command configlist

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="comment 5"
 date="2014-05-24T21:45:55Z"
 content="""
Looks like .exe prefix confuses detection logic. If I remove extension, it does behave as git-annex-shell.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="comment 6"
 date="2014-05-24T21:49:47Z"
 content="""
However, it's still not usable remotely over ssh:

git annex sync kpf

git-annex-shell: git: createProcess: invalid argument (Invalid argument)

  Remote kpf does not have git-annex installed; setting annex-ignore

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="comment 7"
 date="2014-05-24T22:07:15Z"
 content="""
OK thought I'd use it with annex-ignore set, but:

    git annex copy --to=kpf file.jpg
    copy file.jpg (checking kpf...) git-annex-shell: git: createProcess: invalid argument (Invalid argument)
(to kpf...)
    git-annex-shell: git: createProcess: invalid argument (Invalid argument)
    rsync: connection unexpectedly closed (0 bytes received so far) [sender]
    rsync error: error in rsync protocol data stream (code 12) at io.c(226) [sender=3.1.0]

    rsync failed -- run git annex again to resume file transfer
    failed
git-annex: copy: 1 failed

For reference, here's the .git/config section.

I can either set receivepack/uploadpack like below or add /cygdrive/c/Program Files (x86)/Git/libexec/git-core to PATH in cygwin shell.

Also, if I set url to kp:/cygdrive/f/backup/f, git complains that it's not a git repository (msysgit doesn't understand cygwin stuff?)

    [remote \"kpf\"]
            url = kp:f:/backup/ph
            fetch = +refs/heads/*:refs/remotes/kpf/*
            receivepack = git receive-pack
            uploadpack = git upload-pack
            annex-uuid = 99c498fc-e38e-11e3-8bb0-5368a291bd90
            annex-ignore = true

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog"
 nickname="Michael"
 subject="comment 8"
 date="2014-05-24T22:58:16Z"
 content="""
Another data point.  If I install cygwin's git (so references like kp:/cygdrive/f would be understood) and remove annex-ignore setting, I get:

    git annex sync kpf
    commit  ok
    pull kpf
    git-annex: Unknown command 'kp'

Looks like hostname (kp) is taken as a command somehow.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnOSgFb3l7nL3Fs7Y9gPGJJjFiV7aJ1tek"
 nickname="Phil"
 subject="comment 9"
 date="2014-07-05T22:29:01Z"
 content="""
Copying the `git-annex.exe` does work but you can also create a symlink and tell Windows to execute symlinks by doing the following.

On the remote machine (the one you are connecting to)

## Create a symlink:

    mklink \"C:\Program Files (x86)\Git\bin\git-annex-shell\" \"C:\Program Files (x86)\Git\bin\git-annex.exe\"

## Tell Windows to run files with no extension:

* Open Environment Variables
* At the end of `PATHEXT` add `;.`

## A problem with rsync

Unfortunately I now get `Access is denied` and `rsync: connection unexpectedly closed`. I think this is due to path issues on Windows. Trying rsync manually I need to use the `/cygdrive/c/rest/of/path` syntax otherwise I get `No such file or directory.`

"""]]


code:
### Please describe the problem.

Changing a file in a repository from a symlink to a normal file causes annex to create an annexed file from that typechange regardless of weather or not it was an annexed file.

### What steps will reproduce the problem?

    git init newrepo
    cd newrepo && git annex init
    touch realfile
    git add .
    git commit -m "added realfile"
    mkdir newdir && cd newdir
    ln -s ../realfile newfile
    git add .
    git commit -m "Added placeholder until we get assets from designers"
    rm newfile
    dd bs=1024 count=10000 if=/dev/zero of=newfile
    git add .
    git commit -m "Finally got assets from designers"
    ls -la newfile
    # lrwxrwxrwx 1 user user <date> newfile -> ../.git/annex/objects/XX/XX/UUID/UUID

### What version of git-annex are you using? On what operating system?

git-annex version: 4.20130802

Ubuntu 12.04 LTS

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

# End of transcript or log.
"""]]

> [[fixed|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="arand"
 ip="130.243.226.21"
 subject="comment 1"
 date="2013-08-12T20:19:37Z"
 content="""
It looks like all the \"add\" operations are missing from the log, which makes it a bit hard to follow...
"""]]


code:
[[!comment format=mdwn
 username="http://caust1c.myopenid.com/"
 nickname="asbraithwaite"
 subject="comment 2"
 date="2013-08-12T20:41:34Z"
 content="""
Sorry about that.  I forgot that -a doesn't add files.
"""]]


code:
[[!comment format=mdwn
 username="arand"
 ip="130.243.226.21"
 subject="comment 3"
 date="2013-08-12T20:48:42Z"
 content="""
There are still no git-annex actions in the log, except for the init. So the end result for ls -la is not possible with those commands. You may want to try to reproduce it again and write down the exact command sequence used ;)
"""]]


code:
[[!comment format=mdwn
 username="http://caust1c.myopenid.com/"
 nickname="asbraithwaite"
 subject="comment 4"
 date="2013-08-12T21:07:45Z"
 content="""
@arand: That's precisely the problem!  I think you'll find if you run those commands exactly, that's what you'll get.

It's got something to do with the git annex pre-commit hook that runs and checks for typechanges.

I shortened the output on the last command to be less verbose, but if the full output is needed, I'll add it.
"""]]


code:
[[!comment format=mdwn
 username="arand"
 ip="130.243.226.21"
 subject="comment 5"
 date="2013-08-12T21:50:47Z"
 content="""
Hmm, I think I see it now, you were missing a `rm` command in the log, and I had no idea the pre-commit hook existed :)

This reproduces the issue, and it seems to indeed be a bug in the git-annex pre-commit handling, nice catch.

<pre>
$ git init temp
Initialized empty Git repository in /home/arand/tmp/temp/.git/
$ cd temp && git annex init
init  ok
(Recording state in git...)
$ ln -s broken link
$ git add .
$ git commit -m\"add broken link\"
[master (root-commit) 8125488] add broken link
 1 file changed, 1 insertion(+)
 create mode 120000 link
$ rm link
$ echo \"actual file\" >link
$ git add .
$ git commit -m\"replace link with real file\"
add link (checksum...) ok
ok
(Recording state in git...)
[master b58b068] replace link with real file
 1 file changed, 1 insertion(+), 1 deletion(-)
$ git diff HEAD~ | cat
diff --git a/link b/link
index 86a410d..4104f6e 120000
--- a/link
+++ b/link
@@ -1 +1 @@
-broken
\ No newline at end of file
+.git/annex/objects/v4/GQ/SHA256E-s12--ef29ded6f5ae80d89a838d37e01ed3efaade7a2994aff87d1100697554b7327b/SHA256E-s12--ef29ded6f5ae80d89a838d37e01ed3efaade7a2994aff87d1100697554b7327b
\ No newline at end of file
$
</pre>

"""]]


code:
[[!comment format=mdwn
 username="http://caust1c.myopenid.com/"
 nickname="asbraithwaite"
 subject="comment 6"
 date="2013-08-12T21:53:17Z"
 content="""
Ah, you're very right.  Thank you.
"""]]


code:
[[!comment format=mdwn
 username="http://caust1c.myopenid.com/"
 nickname="asbraithwaite"
 subject="comment 7"
 date="2013-08-12T21:53:56Z"
 content="""
I updated the issue with the `rm` statement I forgot.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.63"
 subject="comment 8"
 date="2013-08-22T17:25:48Z"
 content="""
Yay for the git-annex community for another fine bug testcase!

The problem is simply that it assumes any typechanged link was an annexed file, and doesn't doublecheck. Fixing that now..
"""]]


code:
[[!comment format=mdwn
 username="http://mey.vn/"
 ip="46.65.14.106"
 subject="git annex status output on connected nodes"
 date="2013-07-25T16:42:04Z"
 content="""
i have now updated the git-annex package to the latest 4.20130723:amd64 still on Debian testing (linux 3.9.8-1) - i see the same issue (while files keep synchronising fine across the various connected nodes).

i have noticed that the same output appears when i issue ```git annex status``` on the other nodes (an Ubuntu 13.04 with kernel 3.9.7 amd64 and a Debian stable with 3.10.1-1 amd64, all with latest Debian 4.20130723:amd64 package).
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4978:f:21a::2"
 subject="comment 2"
 date="2013-07-25T20:12:06Z"
 content="""
Please paste the output of: `git show git-annex:uuid.log`

The file seems to have a lot of garbage in it. The \"merging -- 10.10.10.10__mnt_annex_00_annex/git-annex into git-annex (recovery from race)\" in particular is only ever passed to git as the commit message. It should not be possible for it to appear in a file unless data has somehow become corrupted.

So, I think you should also run `git fsck`. You may also find that it helps to shut down any running git-annex, and move `.git/annex/index` away to a backup location. A corrupted index file would be kind of be the best scenario since git-annex can automatically rebuild it from the git repository..
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4978:f:21a::2"
 subject="comment 4"
 date="2013-07-26T16:52:26Z"
 content="""
The 4201 dangling blobs is a little bit strange, although it could certainly happen in some normal scenarios.
Overall, your repository seems to be ok except for this weird data in the one file. I do not anticipate the extra garbage causing any problems at all.

To track this down, we need to find the commit that added the garbage. One way, assuming you're using indirect mode, is to `git checkout git-annex; git blame uuid.log` and `git show` the commit that added the garbage. If you're using direct mode, you should first `git clone` the repository and do that in the clone.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.246.110"
 subject="comment 6"
 date="2013-07-26T20:59:42Z"
 content="""
I think that the whole `git show $sha` for each commit will provide useful context.

I want to understand what happened here, because there are multiple weird things going on, and it's important to root cause this sort of bug.

What version of git-annex is on the rPi and how was it installed? Is there a simple operation you can do that segfaults every time?
"""]]


code:
[[!comment format=mdwn
 username="http://mey.vn/"
 ip="46.65.14.106"
 subject="comment 7"
 date="2013-07-26T22:31:15Z"
 content="""
thanks Joey. i'm sending the output of ```git show $sha``` separately as it contains some non-public data; according to apt/dpkg logs, the rPi was actually using git-annex:armhf 4.20130521 from Debian testing at the time these commits were done. i did in fact install 4.20130709 and then 4.20130723 a couple of days after the issue showed up, but only yesterday i realised that both versions were segfaulting all the time (just invoking ```git-annex``` would segfault); 4.20130521 works reliably, instead (i have just downgraded to test), so my previous question about an interrupted operation possibly causing the metadata corruption is not valid as the git-annex executable in use at the time of the issue was not segfaulting (at least not under every circumstance as the more recent versions *in my setup* - but i'll dig into this separately - probably some library issue or so).

the only change i can't really trace in terms of timing is when i deleted the remote i had originally set up on my rPi (using encrypted rsync) via the assistant's web interface on my laptop, and set it up again as plain unencrypted ssh remote (bare repo). i'm pretty sure this operation was done the day the commits in the ```git blame``` for ```uuid.log``` were done, but i'm not sure about the time and don't have logs going back to that moment.

i could disconnect all the remotes from my laptop and redo the setup on the rPi to see if i can reproduce the issue, but i would need some pointers on how to revert my main annex on my laptop to the state it was in before i first started using the rPi as remote: can i just ```git reset --hard <some_sha>``` while on the ```git-annex``` branch? the annex is currently in direct mode (and as been so for a few months now).

otherwise, i could easily copy all the data (or a subset, to start with) to a clean folder, set up a fresh annex there and pair the rPi again - if the corruption was due to some issue on the rPi it may not depend on the exact status the annex was at when the metadata corruption happened (although my full setup includes other remotes and i see some back-and-forth of merging going on throughout the history of the annex around the time the metadata issue started, so it may be hard to reproduce the exact same context).
"""]]


code:
[[!comment format=mdwn
 username="http://mey.vn/"
 ip="46.65.14.106"
 subject="comment 7"
 date="2013-07-26T18:11:25Z"
 content="""
thanks; the current ```uuid.log``` file carries content from 8 distinct commits :

    335cb49c  2013-07-22 16:57:32 +0100
    4ec397f7  2013-07-22 16:57:26 +0100
    50542096  2013-07-22 17:19:21 +0100
    621a1a03  2013-07-22 16:57:28 +0100
    63dd4509  2013-07-22 16:57:26 +0100
    8c2d3ff7  2013-07-22 16:57:19 +0100
    9ba4166d  2013-07-22 18:38:29 +0100
    e182ee60  2013-07-22 16:57:26 +0100

would the whole diff for each commit be useful, or just the part related to uuid.log?

all these commits were done by git-annex on my laptop; the earliest of these commits come seconds after two commits done on another node - a raspberry pi running armhf Raspbian, although i now see that ```git-annex``` and ```git-annex-shell``` on this rPi are constantly segfaulting (though i can't understand the reason from the strace output): perhaps a segfault while dealing with a git operation on the rPi started the metadata corruption somehow?

btw - if this could be useful to track down a possible bug in git-annex (although only involving some metadata, not the actual data), i am happy to dig further to provide all the information needed to understand what might have happened, but if it's not something worth investigating further in the bigger scheme of things, from my point of view i'm ok with restarting a fresh annex with content moved from this annex, and pristine metadata :)

"""]]


code:
[[!comment format=mdwn
 username="http://mey.vn/"
 ip="46.65.14.106"
 subject="comment 8"
 date="2013-07-26T22:38:51Z"
 content="""
forgot to mention - git-annex:armhf 4.20130521 was installed on the rPi via apt-get after having added testing as extra source besides the base stable of Raspbian; the other two versions (the segfaulting ones) were installed via ```dpkg -i``` on the packages downloaded directly from packages.d.o.
"""]]


code:
### Please describe the problem.
Within the walkthrough the deprecated annex.numcopies are still used. At least it should be added that "git annex numcopies x" can be used to define it globally without the need for a .gitattribute file. And use .gitattribute only for per directory and fine grained control.
http://git-annex.branchable.com/walkthrough/backups/

### What steps will reproduce the problem?

### What version of git-annex are you using? On what operating system?

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

# End of transcript or log.
"""]]

> [[fixed|done]] --[[Joey]]


code:
### Please describe the problem.

When git annex is used on a encfs mount, ``.fuse_hiddenXXXX`` files are left behind in ``.git/annex/objects/*/*/`` when I ``git annex move`` files off the encfs

### What steps will reproduce the problem?

    mkdir normal plain enc
    encfs -o kernel_cache "$(realpath enc)" "$(realpath plain)"

    cd normal
    git init
    git annex init normal

    cd ../plain
    git init
    git annex init enc
    dd if=/dev/urandom bs=1M count=11 of=11m
    git annex add 11m
    git remote add normal "$(realpath ../normal)"
    du -hscL *
        << 11MB-ish >>
    du -hsc .git/annex/objects
        << 11MB-ish >>
    git annex move --to=normal 11m
    du -hscL *
        << 0 >>
    du -hsc .git/annex/objects
        << 11MB-ish but should be 0-ish >>
    tree -a .git/annex/objects
        << ??/??/SHA1-*/.fuse_hidden??????? >>
    lsof .git/annex/objects/??/??/SHA*/.fuse_hidden*
        << shows no programs have this file open >>

### What version of git-annex are you using? On what operating system?

5.20131221 on debian unstable (installed by aptitude from debian/main)

### Please provide any additional information below.

I did a couple quick internet searches on these .fuse_hidden files, and (if a random person on the internet is correct) they are created by enfs (or maybe fuse itself) when a file is unlinked while open. It is supposed to be deleted when the file is closed.

Possibilities come to mind:

*    There's a bug in encfs/fuse
*    git annex fails to close the file, and whatever cleanup happens when the linux kernel closes git annex fails to trigger encfs/fuse's real unlinking/cleanup thing

I'm happy to help with further testing, experimenting, etc. I'm good at git, the commandline, programming, etc but I don't know Haskell.

- Jason <jason@jasonwoof.com>


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 1"
 date="2013-12-31T19:35:20Z"
 content="""
When git-annex is deleting an object's content, it opens the object, locks it, unlinks it, and closes it. This is necessary in order to avoid various race conditions that could cause data loss (ie, another repository might be in the process of verifying that the object that is being deleted is present).

I suspect that is what causes the problem with encfs. Since it is entirely a valid POSIX operation to do this, and since not doing it could cause data loss in some situations, I am very reluctant to change anything in git-annex in this area.

It's unfortunate that encfs seems to cause so many problems for POSIX filesystem semantics. Luckily, there are other ways to encrypt filesystems that do not suffer from such problems.
"""]]


code:
### Please describe the problem.
Filenames for the *include* statement for [preferred content](http://git-annex.branchable.com/preferred_content/) can not contain spaces.

### What steps will reproduce the problem?

* Create an annex repo
* Run `git annex vicfg`
* Enter expression *include='pictures/dir with spaces'*
* git annex complains: # ** Parse error in next line: Parse failure: near "with" Parse failure: near "spaces'"

(The *'* is interpreted as part of the filepath.)

### What version of git-annex are you using? On what operating system?

I am using the current binaries from Debian stable amd64.

<pre>
git-annex version: 5.20140320~bpo70+1
build flags: Assistant Webapp Pairing S3 Inotify DBus XMPP Feeds Quvi TDFA
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SHA256 SHA1 SHA512 SHA224 SHA384 WORM URL
remote types: git gcrypt S3 bup directory rsync web tahoe glacier hook external
local repository version: 5
supported repository version: 5
upgrade supported from repository versions: 0 1 2 4
</pre>

### Please provide any additional information below.

The only workaround I found is to use a glob for the filepath which only works for the first space: *include='pictures/dir\*'*.

[[!tag confirmed]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.244"
 subject="comment 1"
 date="2014-04-02T18:45:36Z"
 content="""
A better workaround is:

include='pictures/dir?with?spaces'

Tokenizing text with embedded quotes is a bit of a PITA, certianly doable I suppose..
"""]]


code:
[[!comment format=mdwn
 username="http://ypid.wordpress.com/"
 ip="213.153.84.215"
 subject="comment 2"
 date="2014-04-02T21:29:54Z"
 content="""
Thanks for your workaround ... Works for me after removing the single quote signs.

With this workaround on hand one could include/match any filename. Works for me ;)
"""]]


code:
### Please describe the problem.

This is a followup from the discussion on https://git-annex.branchable.com/forum/Standard_groups__47__preferred_contents/ where I unfortunately did not get a complete answer.
I don't know if it is really a bug but at least it does not work as I would expect and the documentation provides no clear discussion on that.

Now to the problem:
My annex is in "manual" mode (or equivalently "exclude="*" and present" or an expression which contains "present".
Then I get a file using "git annex get file".
I would expect that this file is now synced because it is "present".
But it is not. When I change the file it is synced to the remotes. This is what it should be.
However, when a remote changes that file, the content is NOT synced, the file is silently dropped.

Similarly, when I get a complete directory tree in manual mode, I would expect that it is synced. That means, when a remote adds a file or changes a file in that directory, it is also synced to the local machine. But it is not. If it is changed, it is silently dropped (as written above). If a file is added, only the metadata is added but the content is not synced.

### What steps will reproduce the problem?

 - Create a file 'file' on the server, git annex add/sync etc.
 - On the client: git annex wanted here 'exclude="*" and present'
 - On the client: git annex get file . The file is now present on the client
 - Change the file on the server, git annex sync
 - git annex sync --content on the client
 - Result: File is dropped again on client

Similarly for directories:

 - Create a (sub-)directory 'subdir' with files and sync everything
 - On the client: git annex get subdir . The subdirectory is now present, all files under it downloaded.
 - On the server create a new file in 'subdir' and git annex add; git annex sync --content
 - git annex sync --content on the client
 - Result: Content of the files is not synced to client

### What version of git-annex are you using? On what operating system?

    git-annex version: 5.20140717-g5a7d4ff
    build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV DNS Feeds Quvi TDFA CryptoHash
    key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
    remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier ddar hook external


code:
Dear All,

thank you for this wonderful tool!

I am having an issue when I try to commit a normal link

diokletian*194-> mkdir test

diokletian*195-> cd test

diokletian*196-> git init

Initialized empty Git repository in /home/henrus/test/.git/

diokletian*197-> git annex init new

init new [master (root-commit) 49f5f91] git-annex setup

 1 files changed, 1 insertions(+), 0 deletions(-)

 create mode 100644 .gitattributes

[master 76496ff] git annex init

 1 files changed, 1 insertions(+), 0 deletions(-)

 create mode 100644 .git-annex/uuid.log

ok

diokletian*198-> mkdir subdir

diokletian*199-> ln -s subdir link

diokletian*200-> git add link

diokletian*201-> git commit -m "ok"

[master f12f62d] ok

 1 files changed, 1 insertions(+), 0 deletions(-)

 create mode 120000 link

diokletian*202-> ln -s subdir/ link2

diokletian*203-> git add link2

diokletian*204-> git commit -m "not ok"

git-annex: Prelude.head: empty list

The trailing slash seems to make a difference!

Best Regards,

Henrik

> Thanks for the bug report. This is fixed in 0.17. --[[Joey]]  [[!tag done]]


code:
On several of my repos, the upgrade to v3 seemed to take forever. A Crl-C followed by another "git annex upgrade" "solved" the problem in some cases. Sometimes, I had to also delete the .git/annex/journal dir to have the upgrade. I didn't notice anything special about the non-working repos to help diagnose the problem.

[[!tag done]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2011-07-04T22:58:46Z"
 content="""
Well if it happens again why don't you use `ps` or `strace` to see what it's doing.
"""]]


code:
[[!comment format=mdwn
 username="https://lithitux.org/openidserver/users/pavel"
 nickname="pavel"
 subject=""Me too""
 date="2011-07-05T15:54:19Z"
 content="""
I've also seen this apparent hang during upgrade to v3. A few more details:

The annex in question has just under 18k files (and hence that many log files), which can slow down directory operations when they're all in the same place (like, for example, .git/annex/journal).

git-annex uses virtually no CPU time and disk IO when it's hanging like this; the first time it happened, 'ps' showed three defunct git processes, with two \"git-annex\" processes and three \"git\" procs:

 * git --git-dir=/mnt/annex/.git --work-tree=/mnt/annex cat-file --batch
 * git --git-dir=/mnt/annex/.git --work-tree=/mnt/annex hash-object -w --stdin-paths
 * git --git-dir=/mnt/annex/.git --work-tree=/mnt/annex update-index -z --index-info

I Ctrl+C'd that and tried again, but it hung again -- this time without the defunct gits.

An strace of the process and its children at the time of hang can be found at http://pastebin.com/4kNh4zEJ . It showed somewhat weird behaviour: When I attached with strace, it would scroll through a whole bunch of syscalls making up the open-fstat-read-close-write loop on .git/annex/journal files, but then would block on a write (sorry, don't have that in my scrollback any more so can't give more details) until I Ctrl+C'd strace; when attaching again, it would again scroll through the syscalls for a second or so and then hang with no output.

Ultimately I detached/reattached with strace about two dozen times and that caused it (?) to finish the upgrade; not really sure how to explain it, but it seems like too much of a timing coincidence.

"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 3"
 date="2011-07-05T17:31:22Z"
 content="""
I've seen this kind of piping stall that is unblocked by strace before. It can vary with versions of GHC, so it would be good to know what version built git-annex (and on what OS version). I filed a bug report upstream before at <http://bugs.debian.org/624389>.

I really need a full strace -f from the top, or at least a complete `strace -o log` of git-annex from one hang through to another hang. The strace you pastebinned does not seem complete. If I can work out which specific git command is being written to when it hangs I can lift the writing out into a separate thread or process to fix it.

@pavel, you mentioned three defunct git processes, and then showed ps output for 3 git processes. Were there 6 git processes in total? And then when you ran it again you said there were no defunct gits -- where the other 3 git processes running once again?

As best I can make out from the (apparently) running git processes, it seems like the journal files for the upgrade had all been written, and the hang occurred when staging them all into the index in preparation for a commit. I have committed a change that lifts the code that does that write out into a new process, which, if I am guessing right on the limited info I have, will avoid the hang.

However, since I can't reproduce it, even when I put 200 thousand files in the journal and have git-annex process them, I can't be sure.
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 4"
 date="2011-07-05T18:37:21Z"
 content="""
I've managed to reproduce this and confirmed my fix works.
"""]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 5"
 date="2011-07-05T19:06:48Z"
 content="""
By the way, the original bug reporter mentioned deleting .git/annex/journal. This is not recommended, and doing it during an upgrade can result in git-annex losing location tracking information. You should probably run `git annex fsck` or reset to the old git tree (and `git config annex.version 2`) and upgrade again.
"""]]


code:
[[!comment format=mdwn
 username="https://lithitux.org/openidserver/users/pavel"
 nickname="pavel"
 subject="comment 6"
 date="2011-07-06T08:14:26Z"
 content="""
Ah, great, thanks very much for the quick fix!

Yes, when I mentioned three defunct git processes, there were three processes shown as \"git [defunct]\", plus the three git processes I listed, plus two \"git-annex\" processes. Upon cancel/resume, there were no defunct git processes when I checked, but by the time I found the bug report on the forum and commented I'd already successfully upgraded by annex (by repeatedly attaching strace) and couldn't really easily get at either additional 'ps' info or a fuller strace than what I posted (that was just the log from one of the attach/detach cycles), so it's a relief you managed to pinpoint the problem.
"""]]


code:
### Please describe the problem.
When my android phone tries to decode files downloaded from a ssh remote using shared encryption, gpg errors occur.

### What steps will reproduce the problem?
Setup is very similar to my other bug report in <https://git-annex.branchable.com/bugs/problems_with_android_and_xmpp/>.
Only difference is the location of the annex on the android side.
I have put it on the /data mount which uses ext4 to avoid the /sdcard fuse mount, which does not handle symlinks.

1) setup git annex via webapp on laptop:

* local annex

* remote annex via ssh with shared encryption (tried two different servers, one with and one without git-annex installed)

* share with my devices using jabber.me account

2) setup git annex via webapp on android:

* local annex in /data/data/ga.androidterm/annex (ext filesystem)

* share with my devices using jabber.me account

* ssh remote is automatically added via XMPP

3) add file to annex on linux, which gets uploaded to the ssh remote

4) symlink for file is created on phone and data downloaded, but never decrypted (see logs below)

### What version of git-annex are you using? On what operating system?
Ubunut Linux 12.04 with git-annex version:

* 5.20140127.1 (from PPA)

Android 4.2 (rooted) with git-annex version:

* 5.20140211-g556cfeb (from autobuilds)

### Please provide any additional information below.
full logs:

* linux: <http://pastebin.ca/2639929>

* android: <http://pastebin.ca/2639945>

most interesting parts:
[[!format sh """
#
# android:
#
gpg: can't open `/usr/local/share/gnupg/options.skel': No such file or directory
gpg: DBG: locking for `/sdcard/git-annex.home/.gnupg/secring.gpg.lock' done via O_EXCL
gpg: DBG: locking for `/sdcard/git-annex.home/.gnupg/pubring.gpg.lock' done via O_EXCL
gpg: decryption failed: bad key

# followed by just the last line for all further attemps
gpg: decryption failed: bad key

# gpg it self seems to be fine
root@android:/data/data/ga.androidterm # ./bin/gpg --version -v
gpg (GnuPG) 1.4.15
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: ~/.gnupg
Supported algorithms:
Pubkey: RSA, RSA-E, RSA-S, ELG-E, DSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB
"""]]


code:
[[!comment format=mdwn
 username="https://openid.stackexchange.com/user/3f8a1927-744c-4409-8425-48fb3c86672f"
 nickname="kim"
 subject="comment 1"
 date="2014-02-14T19:16:02Z"
 content="""
I am experiencing the exaxact same issue on android 4.3 with the current git-annex version (5.20140211)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm78jq1Uo-ZbyOPG3diJUWVvEiM0kyAcvk"
 nickname="Dorian"
 subject="any ideas or questions?"
 date="2014-02-25T13:41:16Z"
 content="""
Hey Joey,

I was wondering if you had any idea how we could fix this problem or if you need further information on this.
Any response would be appreciated.

Thanks for your great work on git-annex!

Cheers,
Dorian
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.146"
 subject="comment 3"
 date="2014-02-26T18:34:18Z"
 content="""
AFAICS, the messages about locking are a red herring, since shared encryption is being used, the public and secret key rings are not used.

This problem seems to be explained here: <http://crypto.stackexchange.com/questions/1468/how-does-gpg-verify-succesful-decryption>

It seems there must be a problem with the symmetric key used for shared encryption. Either the repository somehow has the wrong key in it, or the key is not extracted from the repository correctly somehow, or it's not fed into gpg correctly somehow.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm78jq1Uo-ZbyOPG3diJUWVvEiM0kyAcvk"
 nickname="Dorian"
 subject="comment 4"
 date="2014-03-03T14:05:35Z"
 content="""
Is it correct, that the shared key is transfered through the git repo, meaning in this case XMPP?

Then I guess this might simply be the problem of unreliable XMPP.
Same problem as here: <http://git-annex.branchable.com/bugs/problems_with_android_and_xmpp/>

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.146"
 subject="comment 5"
 date="2014-03-05T20:49:20Z"
 content="""
Yes, the key is in the git repository.

While XMPP is an unreliable transport, it either manages to git pull over it, or it fails, git does not allow partial or corrupt pulls.
"""]]


code:
### Please describe the problem.
When trying to sync my android phone with my linux laptop using the git annex assistant and XMPP no files are transferred.

### What steps will reproduce the problem?
1) setup git annex via webapp on laptop:

* local annex

* remote annex via ssh with shared encryption (tried two different servers, one with and one without git-annex installed)

* share with my devices using jabber.me account

2) setup git annex via webapp on android:

* local annex in /sdcard/annex (fuse filesystem without symlink support)

* share with my devices using jabber.me account

* ssh remote is automatically added via XMPP

3) add files to annex on linux, they get uploaded to the ssh remote

4) wait forever for any files from linux to download to phone

5) add files to annex on phone, they get uploaded to the ssh remote

4) wait forever for any files from phone to download to linux

### What version of git-annex are you using? On what operating system?
Ubunut Linux 12.04 with git-annex version:

* 5.20140127.1 (from PPA)

Android 4.2 (rooted) tried with git-annex versions:

* 5.20140209 (from http://downloads.kitenet.net/git-annex/android/current/4.0/)

* 5.20140211-g556cfeb (from autobuilds)

### Please provide any additional information below.
full logs:

(these do not show the uploads to the ssh remote, because I restarted to get clean and short logs, but the files are on the server and can be dropped and restored on the linux side manually)

* linux: <http://pastebin.ca/2639948>

* android: <http://pastebin.ca/2639952>

most interesting parts:
[[!format sh """
#
# linux:
#

[2014-02-13 13:11:27 CET] XMPPClient: Pairing with dorian in progress
[2014-02-13 13:11:28 CET] XMPPSendPack: Syncing with dorian
To xmpp::dorian@jabber.me
 * [new branch]      git-annex -> refs/synced/4ce7576f-6f02-4657-bab5-2f4c4a564ee7/ZG9yaWFuQGphYmJlci5tZQ==/git-annex
 * [new branch]      annex/direct/master -> refs/synced/4ce7576f-6f02-4657-bab5-2f4c4a564ee7/ZG9yaWFuQGphYmJlci5tZQ==/annex/direct/master
[2014-02-13 13:11:29 CET] XMPPSendPack: Syncing with dorian
Everything up-to-date
[2014-02-13 13:12:21 CET] XMPPSendPack: Syncing with dorian
Everything up-to-date
[2014-02-13 13:12:21 CET] XMPPSendPack: Syncing with dorian
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

#
# android:
#
[2014-02-13 13:16:25 CET] XMPPClient: sending: Pushing "d29" (ReceivePackOutput 2 "<elided>")
[2014-02-13 13:16:25 CET] XMPPClient: to client: d6/tigase-14134
[2014-02-13 13:18:22 CET] XMPPClient: received: ["Unknown message"]
[2014-02-13 13:18:25 CET] XMPPReceivePack: timeout waiting for git send-pack output via XMPP
fatal: The remote end hung up unexpectedly
[2014-02-13 13:18:25 CET] XMPPReceivePack: finished running push Pushing "d29" (StartingPush (UUID "4ce7576f-6f02-4657-bab5-2f4c4a564ee7")) True
[2014-02-13 13:18:25 CET] XMPPClient: sending: Pushing "d29" (ReceivePackDone (ExitFailure 128))
[2014-02-13 13:18:25 CET] XMPPClient: to client: d6/tigase-14134

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm78jq1Uo-ZbyOPG3diJUWVvEiM0kyAcvk"
 nickname="Dorian"
 subject="further information"
 date="2014-02-13T13:33:49Z"
 content="""
I just tried moving the annex on the android phone to a different location, because I wasn't sure how git annex handles the /sdcard's fuse file system without symlinks...
So with the annex on the /data mount using ext I got a little further (and discovered a different problem <https://git-annex.branchable.com/bugs/problems_with_android_and_gpg/>).

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.146"
 subject="comment 2"
 date="2014-02-26T18:52:17Z"
 content="""
I don't see any indications that this problem is specific to Android.

Relying on git push over XMPP is known to be somewhat fragile, and some XMPP servers are known to do things with the XMPP protcol that make it impossible to reliably do git push over it. [[design/assistant/xmpp]] discusses these problems. Since this seems a basically intractable problem to solve, git-annex will be moving away from XMPP as soon as something better is available. (eg [[design/assistant/telehash]])

The best way to use XMPP currently is as a simple signaling mechanism to tell when a push has been made to a git repository. Your ssh server seems to have an encrypted rsync repository on it, so no git repository. If you can put up a git repository someplace both the android and linux machine can access, I think that you'll find it greatly improves the reliability of the syncing.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm78jq1Uo-ZbyOPG3diJUWVvEiM0kyAcvk"
 nickname="Dorian"
 subject="comment 3"
 date="2014-03-03T14:01:10Z"
 content="""
You're right, with an unencrypted git repo in between it works.
I was hoping to be able to avoid that, though.

My very first try was actually to switch XMPP servers, because I thought they might be the problem. But after trying both servers you mentioned in the assistant and on the instruction pages (Google and jabber.me) I figured it should at least work with one of them...

So if XMPP can actually only be used as a simple signaling mechanism, this should be mentioned explicitly in the \"remote sharing\" and \"share with a friend\" walkthroughs.
After reading/watching them I was under the impression, that after setting up XMPP, you can use any cloud storage to exchange files.
But if I now understand correctly this cloud storage needs to have a full git repo, not only the encrypted files.
And for Android you cannot use a gcrypt remote, so only unencrypted git repos are possible, making any public cloud service unfeasible.

Is my conclusion correct, that this leaves only one option when sharing with Android?
1) unencrypted git repo on trusted/private special remote

Or am I missing something?

Thanks again for this!
"""]]


code:
### Please describe the problem.
Annex errors when copying to glacier.

### What version of git-annex are you using? On what operating system?

OS X 10.9.2 Build 13C64

    git-annex version: 5.20140318-gdcf93d0
    build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV FsEvents XMPP DNS Feeds Quvi TDFA CryptoHash
    key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
    remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier hook external
    local repository version: 5
    supported repository version: 5
    upgrade supported from repository versions: 0 1 2 4

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

> git annex initremote glacier type=glacier encryption=hybrid keyid=E9053BDA datacenter=us-west-1                           ║██████████╠ ∞ ∞
initremote glacier (encryption setup) (hybrid cipher with gpg key B608B8F6E9053BDA) ok
(Recording state in git...)
> git annex copy Cobalt\ Strike\ Tradecraft --to=glacier --debug
[2014-03-27 07:27:39 PDT] read: git ["--git-dir=/Users/akraut/Desktop/annexes/media/.git","--work-tree=/Users/akraut/Desktop/annexes/media","show-ref","git-annex"]
[2014-03-27 07:27:39 PDT] read: git ["--git-dir=/Users/akraut/Desktop/annexes/media/.git","--work-tree=/Users/akraut/Desktop/annexes/media","show-ref","--hash","refs/heads/git-annex"]
[2014-03-27 07:27:39 PDT] read: git ["--git-dir=/Users/akraut/Desktop/annexes/media/.git","--work-tree=/Users/akraut/Desktop/annexes/media","log","refs/heads/git-annex..9f59057d857784e6ae6b3dcd6793092264375913","--oneline","-n1"]
[2014-03-27 07:27:39 PDT] chat: git ["--git-dir=/Users/akraut/Desktop/annexes/media/.git","--work-tree=/Users/akraut/Desktop/annexes/media","cat-file","--batch"]
[2014-03-27 07:27:39 PDT] read: git ["config","--null","--list"]
[2014-03-27 07:27:39 PDT] read: git ["--git-dir=/Users/akraut/Desktop/annexes/media/.git","--work-tree=/Users/akraut/Desktop/annexes/media","ls-files","--cached","-z","--","Cobalt Strike Tradecraft"]
copy Cobalt Strike Tradecraft/Tradecraft__1_of_9____Introduction.mp4 (gpg) [2014-03-27 07:27:39 PDT] chat: gpg ["--quiet","--trust-model","always","--decrypt"]

You need a passphrase to unlock the secret key for
user: "Andrew Mark Kraut <akraut@gmail.com>"
4096-bit ELG-E key, ID 353E49B9, created 2008-11-11 (main key ID E9053BDA)

(checking glacier...) [2014-03-27 07:27:46 PDT] read: glacier ["--region=us-west-1","archive","checkpresent","git-annex: Maybe.fromJust: Nothing

# End of transcript or log.
"""]]

> This was a bug introduced last month, it forgot to receord the
> datacenter and vault used when initializing the glacier repository.
>
> I've fixed the bug, but this does not fix repositories created with
> the broken version. I considered just making it use the default
> datacenter and vault for such a repository, but
> a) those may change in the future
> and I don't want to have to worry about breaking such a repository
> going forward and b) someone may have overridden it to use another
> datacenter or vault name and so it shouldn't blindly assume the defaults.
>
> Instead, there's a manual fix up step you need to do. Luckily quite easy.
> For example:
>
>	git annex enableremote myglacier datacenter=us-east-1 vault=myglacier-fae9be57-8eb4-47af-932f-136b9b40e669
>
> The default datacenter is us-east-1, and the default vault name is
> "$remotename-$uuid". So you just have to tell it these values
> once with an enableremote command, and it will then work.

> You don't even need to get the fixed version of git-annex to work
> around the bug this way.. Although it does have better error messages
> too. [[fixed|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmRFKwny4rArBaz-36xTcsJYqKIgdDaw5Q"
 nickname="Andrew"
 subject="comment 1"
 date="2014-03-27T14:46:19Z"
 content="""
I just updated to the latest glacier-cli and boto and have confirmed that those are working properly and that the vault has been created on glacier:
[[!format sh \"\"\"
> glacier --region=us-west-1 vault list
glacier-571d1ec3-8870-46cb-977e-15830a2b474d
\"\"\"]]
"""]]


code:
### Please describe the problem.

git annex now fails to transfer a fail with: `protocol version mismatch -- is your shell clean?`

### What steps will reproduce the problem?

start a transfer, then switch between your wireless and wired connexions (I am using network-manager), then interrupt the transfer with control-c.

### What version of git-annex are you using? On what operating system?

on my side: 5.20140306~bpo70 on debian wheezy amd64

on the other side: 4.20130815 on ubuntu saucy i386

### Please provide any additional information below.

[[!format sh """
anarcat@angela:video$ git annex copy --to t films/foo.mkv
copy films/foo.mkv (checking t...) (to t...)
protocol version mismatch -- is your shell clean?
(see the rsync man page for an explanation)
rsync error: protocol incompatibility (code 2) at compat.c(174) [sender=3.0.9]

  rsync failed -- run git annex again to resume file transfer
failed
git-annex: copy: 1 failed
"""]]

workaround: `cd .git/annex/; mv transfer transfer.old` on the other side.

-- [[anarcat]]

[[!taglink moreinfo]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.191"
 subject="comment 1"
 date="2014-04-20T16:52:15Z"
 content="""
Last time rsync seemed to fail like this it was really the remote git-annex-shell failing on an encoding problem:
[[!commit 0b12db64d834979d49ed378235b0c19b34e4a4d6]]

It seems I would need to see the transfer info files you moved out of the way to say more. Or you could copy the back, reproduce the problem, find the git-annex-shell command that is being run (using --debug), and see if you can run it on the remote system and reproduce the problem there without rsync in the picture, in order to get the actual error message.
"""]]


code:
Installed stand-alone tarball amd64.
I was able to launch webapp. (on laptop)
Created a repository to local home directory.
Then ceated another repository to invite local desktop pc.(this one had all the files i wanted to sync)
Went to dektop and accepted invitation.  But both machines never stopped synching?  and nothing really happened.
so I removed repository on laptop to start fresh.
But now webapp does nothing.  I removed and re-installed a few times but still nothing.
Only as superuser will the webapp attempt to load but fails because it is super user running.
As far as version of git-annex...  it prompted to upgrade and i think i saw a 5 in there somewhere.

and since it won't load anymore i guess there is no log.

[[!tag moreinfo]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="71.80.94.56"
 subject="comment 1"
 date="2014-02-07T19:14:47Z"
 content="""
There's a bit of a dearth of information here. I think I need to know:

* What the version is (output of `git annex version`)

* What happens when you run `git annex webapp` (\"does nothing\" is very vague. Does it sit there forever without returning to the prompt? Print an error message? Open a web browser that's not usable somehow?)

You might take a look at the file `~/.config/git-annex/autostart` and remove any repository you deleted from there, although normally git-annex will ignore entries that no longer exist.
"""]]


code:
#What steps will reproduce the problem?

Running assistant in foreground on one repository that is paired to another repository, than killing it using CTRL-C and restarting it

#What is the expected output? What do you see instead?

I get messages like:

    (Recording state in git...)
    # Auf Zweig master
    # Änderungen, die nicht zum Eintragen bereitgestellt sind:
    #   (benutze "git add/rm <Datei>..." zum Bereitstellen)
    #   (benutze "git checkout -- <Datei>..." um die Änderungen im Arbeitsverzeichnis zu verwerfen)
    #
    #       gelöscht:    "path/to/file"
    #
    keine Änderungen zum Eintragen hinzugefügt (benutze "git add" und/oder "git commit -a")
    Total 0 (delta 0), reused 0 (delta 0)
    To ssh://stormking@git-annex-volyova-stormking/data/repository/
       4e2c631..911b80c  git-annex -> synced/git-annex

    Already up-to-date.

Sorry for the german language, I'll try to reproduce it in english, later.
After that, the symlinks for the file in the repository are gone. I can get
them back by reverting the commit but things like that make me very nervous.

#What version of git-annex are you using? On what operating system?

3.20130102 on Arch Linux x64

[[!tag /design/assistant moreinfo]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.7.238"
 subject="comment 1"
 date="2013-01-17T18:30:06Z"
 content="""
AFAICS, the German message is the equivilant of `git status` when a file has been deleted:

<pre>
# On branch master
# Changes not staged for commit:
#   (use \"git add/rm <file>...\" to update what will be committed)
#   (use \"git checkout -- <file>...\" to discard changes in working directory)
#
#	deleted:    bigfile
#
no changes added to commit (use \"git add\" and/or \"git commit -a\")
</pre>

There's no indication here of when the file was deleted, or what deleted it. I need a way to reproduce this to be able to help. At this point it's not clear if the file is deleted by the old assistant that you ctrl-c'd, or if it was deleted while the assistant was not running, or if it somehow gets deleted when the new assistant is started up.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnjjCyhVEcTRM5m4iIBqL3ZCooPx7ZYB_E"
 nickname="Marcus"
 subject="comment 2"
 date="2013-01-18T09:08:05Z"
 content="""
>  At this point it's not clear if the file is deleted by the old assistant that you ctrl-c'd, or if it was deleted while the assistant was not running, or if it somehow gets deleted when the new assistant is started up.

So how can I help to track this down? So far I can only rule out the file being deleted by some other process.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.3.194"
 subject="comment 3"
 date="2013-01-18T20:18:36Z"
 content="""
If you can reproduce the problem, you should be able to see if the file is there when you've ctrl-c the assistant.
"""]]


code:
### Please describe the problem.
In latest git-annex version for windows (4.2130723), after a fresh clone, git annex sync considers dummy symlinks as recently modified files and commits the content.

Alternatively, to try and avoid this problem, I tried manually merging branches (git-annex and master) but if I do that, I can't retrieve files anymore. That is, `git annex get .` downloads files but doesn't replace the dummy symlinks (I am guessing the same as `http://git-annex.branchable.com/bugs/windows_port_-_can__39__t_directly_access_files/`. Tell me if this is a different bug, I'll file a new bug report)

### What steps will reproduce the problem?
Create a new repository (on windows or on linux), create a file and commit it.

Clone this repository (on windows), then `git annex init` it. The file is here, containing the path to the real file (like a symlink but the crippled filesystem's version).

At this point, if you perform `git annex sync`, git-annex thinks the dummy symlink is the new content of the file and commits it.

### What version of git-annex are you using? On what operating system?
This problem occurs on git-annex for windows version 4.20130723 (the latest version as of now) although it worked well in version 4.20130709.

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
$ mkdir test1

$ cd test1

$ git init
Initialized empty Git repository in c:/Users/raz/test1/.git/

$ git annex init test1
init test1
  Detected a crippled filesystem.

  Enabling direct mode.

  Detected a filesystem without fifo support.

  Disabling ssh connection caching.
ok
(Recording state in git...)

$ echo "This is the content of the file" > file.txt

$ git annex add
add file.txt (checksum...) ok
(Recording state in git...)

$ git annex sync
commit
ok
git-annex: no branch is checked out

$ cat file.txt
This is the content of the file

$ cd ..

$ git clone test1 test2
Cloning into 'test2'...
done.

$ cd test2

$ git annex init test2
init test2
  Detected a crippled filesystem.

  Enabling direct mode.

  Detected a filesystem without fifo support.

  Disabling ssh connection caching.
ok
(Recording state in git...)

$ cat file.txt        # File is only a dummy file, it contains the path to the real file (which doesn't yet exist here, this is expected behavior)
.git/annex/objects/33/43/SHA256E-s32--a50017c6136930a142cfca6ee34b700d96dcf0ba59cf7007c27c2924f80dfa7a.txt/SHA256E-s32--a50017c6136930a142cfca6ee34b700d96dcf0ba59cf7007c27c2924f80dfa7a.txt

$ git annex sync
(merging origin/git-annex into git-annex...)
(Recording state in git...)
add file.txt (checksum...) ok                # ??? The dummy file is added to the index -> shouldn't happen
(Recording state in git...)
commit
(Recording state in git...)
ok
pull origin
ok
push origin
Counting objects: 26, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (14/14), done.
Writing objects: 100% (19/19), 1.78 KiB | 0 bytes/s, done.
Total 19 (delta 2), reused 0 (delta 0)
To c:/Users/Renaud Casenave-Pere/test1
 * [new branch]      git-annex -> synced/git-annex
 * [new branch]      master -> synced/master
ok

$ git annex whereis                     # File should be in origin repository, not here
whereis file.txt (1 copy)
        e6e1c558-7127-4ffa-a79b-2161b44ec44b -- here (test2)
ok

$ cat file.txt                          # The committed content, discarding the real content
.git/annex/objects/33/43/SHA256E-s32--a50017c6136930a142cfca6ee34b700d96dcf0ba59cf7007c27c2924f80dfa7a.txt/SHA256E-s32--a50017c6136930a142cfca6ee34b700d96dcf0ba59cf7007c27c2924f80dfa7a.txt

# End of transcript or log.
"""]]

Tell me if you need further information.

> [[fixed|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.21"
 subject="re: workaround"
 date="2013-07-30T19:17:11Z"
 content="""
The problem with manually merging the branches and not using sync is that the file mappings get out of date. You should be able to correct them by running `git annex fsck`
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.21"
 subject="comment 2"
 date="2013-07-30T19:55:04Z"
 content="""
This reversion affects direct mode on FAT, not just on Windows! It was probably caused by commit ecdfa40cbea1ae213ab84913d8f011027967a610 or commit ae341c1a37eecc1724517e3e025d144badb5abfe. Investigating.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.21"
 subject="comment 3"
 date="2013-07-30T20:07:09Z"
 content="""
Yeah, I inverted some logic. This also affects, for example, git-annex on Android. Sigh. Sorry about this.

I suppose that, if you run into this bug, the best way to fix up after it is to `git-revert -n` the bad commit that sync made. Then run `git annex sync` with the fixed git-annex to commit the reversion.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.21"
 subject="fixed"
 date="2013-07-30T20:11:19Z"
 content="""
Seems like the test suite should have caught this on Windows. Unfortunately, the part of the test suite that tests sync is commented out due to it not working at all from within the test suite for reasons I don't understand. Pity.
"""]]