code:
### Please describe the problem.

#### git annex unused fails with the message:

...
   86731   SHA256E-s13243--a6c5fe9710975e0b0ca01ec0b7eca6068a01b1b1c06adbd8ad7921e26d3b076d.h

   86732   SHA256E-s12872--f50b73d313a116ea21649d684f601840dbc8ed3c264165dd77fa5d54a1c57464.c

   86733   SHA256E-s6557264--6b502d56378b9919fd3c54c3bb2dc2906a326be8e9d477a8f3518419929f4706.dll

   86734   SHA256E-s141--c6227fe715dab2b4447b6e43af2b170ac66d82043aa81a14013c56be922e11db.out

   86735   SHA256E-s6564--ec4bcd833d071fff7d8cc81b908558acdeae2c9ba62f7d65b584effe6b36a8c6.c

git-annex: /bluepool/data/.git/annex/unused32594.tmp: commitBuffer: invalid argument (invalid character)
failed

git-annex: unused: 1 failed

#### A subsequent git annex dropunused does not work (there are 86735 files unused). A simple:
"git annex dropunused 1-10" produces:

git-annex: 1 not valid (run git annex unused for list)

#### It seems that the list of unused files was not saved (committed)

### What steps will reproduce the problem?

I suspect it can be reproduced by having files with non-english characters.
This should be the sequence of commands to reproduce if one knew the invalid character:

git init .

git annex init .

touch "Invalid character in file name"

git annex add "Invalid character in file name"

git commit

git rm "Invalid character in file name"

git commit

git unused

However, I usually have "git annex watch" running so this might also have an influence on how (maybe not-escaped) filenames are added.

### What version of git-annex are you using? On what operating system?

git annex standalone:

git-annex version: 5.20140306-g6e2e021

build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP Feeds Quvi TDFA CryptoHash

key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL

remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier hook external

local repository version: 5

supported repository version: 5

upgrade supported from repository versions: 0 1 2 4

Ubuntu Raring: Linux i3 3.11.4-031104-generic

### Please provide any additional information below.

I guess I could provide the full output of git annex unused but I would rather not put it into this bug report for privacy reasons.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

# End of transcript or log.
"""]]

> [[closing|done]], would still be nice to have verification my fix worked.
> --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.154"
 subject="comment 1"
 date="2014-03-15T16:01:53Z"
 content="""
I see the problem, although I have not quite managed to reproduce it. I think this would probably only happen if you have the unusual character in the extension of the file (or are using the WORM backend).

I've put a fix in git. It will be available in the next autobuild, in half an hour or so.
"""]]

repo_name: kakafarm/git

license: unknown


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmp1ThsNNAbSn46ju-gwFELfStlhl8usJo"
 nickname="donkeyicydragon"
 subject="Problem still occurs but maybe my update did not work"
 date="2014-03-16T18:01:24Z"
 content="""
Hi Joey, thanks for the quick response.

I tried to update and retry my \"git annex unused\" but it gave the same error. Maybe my updating did not work (see below). But is there a way I can retry it without running \"git annex unused\" again? The problem is that each run takes several hours with my amount of files.
It would be nice, if git annex recognized some error in the last approach and recovered.
The tmp files that are created by \"git annex unused\" are still in .git/annex/.

Regarding the updating:
I tried to obtain the autobuild version but on this website I could not find a link to download the latest autobuild. I could only find the build reports. But I used the git annex installer script from https://github.com/zerodogg/scriptbucket/blob/master/gitannex-install
and it apparently knows the location of the daily build. Is this the autobuild or is git annex autobuilt more often?

This is my version after installing it yesterday evening approx. 2h hours after your answer to the bug description.

git-annex version: 5.20140314-g48e0f18
build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV Inotify DBus XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier hook external
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.154"
 subject="comment 3"
 date="2014-03-17T19:28:05Z"
 content="""
No, your version is from before I fixed it.

You can get an autobuild from [[install/Linux_standalone]]
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmp1ThsNNAbSn46ju-gwFELfStlhl8usJo"
 nickname="donkeyicydragon"
 subject="Maybe the autobuild is broken?"
 date="2014-03-17T20:32:19Z"
 content="""
The latest autobuild log for amd64 says:
Starting at: Fri Mar 14 21:42:07 UTC 2014

I downloaded the amd64 autobuild from https://git-annex.branchable.com/install/Linux_standalone/

and it indeed is from Friday.

Cheers,
Marek
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.154"
 subject="comment 5"
 date="2014-03-18T19:07:25Z"
 content="""
cron was not running them for some reason, but they are up-to-date now.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmp1ThsNNAbSn46ju-gwFELfStlhl8usJo"
 nickname="donkeyicydragon"
 subject="Your changes to git annex worked"
 date="2014-03-19T21:27:39Z"
 content="""
Thanks a lot.

BTW, will you start a new funding round?
"""]]


code:
I upgraded another one of my git-annex clones.  The upgrade worked fine (i.e.
according to the manual) on two other clones before, but this time something is
different.

After 'git pull' and 'git annex upgrade', which took a long time and seemed to
have succeeded, there are no staged changes in git.  Instead there are lots of
untracked directories in .git-annex.  Aside from that, nothing seems to be
wrong.

At the time I had git-annex version 0.20110329 and I've been using the SHA1
backend since version 1.

> Yes, I agree with Jimmy, it's the same bug. So I'll be closing this one.
> Please keep us informed how the workaround committed to git-annex
> yesterday for the case insensativity issue works out. [[dup|done]]
> --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 1"
 date="2011-04-03T02:26:20Z"
 content="""
I'm not sure how this happened, as far as I can see, and based on my testing, `git annex upgrade` does stage the location log files. OTOH, I vaguely rememeber needing to stage some of them when I was doing my own upgrades, but that was a while ago, and I don't remember the details.

Your upgrade seems to have gone ok from the file lists you sent, so you can just: `git add .git-annex; git commit`
"""]]


code:
[[!comment format=mdwn
 username="gernot"
 ip="213.168.117.192"
 subject="comment 2"
 date="2011-04-03T15:35:52Z"
 content="""
'git add .git-annex' didn't do anything.  That's when I noticed that this
repository is on a case-insensitive HFS+ file system.

So, if I get this right it's not a new bug, but similar to this situation:
[[git-annex_directory_hashing_problems_on_osx]]

Assuming that it was the file system's fault, I went ahead and upgraded yet
another clone.  That one (on an ext3 file system) had neither staged changes
nor left-over untracked files.  Everything seems to just have fallen right into
place.  Is that possible or still weird?

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 3"
 date="2011-04-03T16:05:39Z"
 content="""
Yes you seem to have come across the same bug that I had initially reported :P
"""]]


code:
## What steps will reproduce the problem?

After the assistant idles for a while, it queues many transfers to remotes configured annex-ignore=true

##What is the expected output? What do you see instead?

No attempts to upload to repos marked 'annex-ignore=true'

Instead I see many queued transfers to the remote 'origin'

[Screenshot](https://www.evernote.com/shard/s1/sh/ea0de76c-4b68-4266-b9f6-8a9c343997b6/72baab4a3ce73b0915b151829bbeaf75/res/8a8ab1fb-3173-47ea-875d-b0e320cb827b/skitch.png)

##What version of git-annex are you using? On what operating system?

    %  > git annex version
    git-annex version: 4.20130315
    local repository version: 3
    default repository version: 3
    supported repository versions: 3 4
    upgrade supported from repository versions: 0 1 2
    build flags: Assistant Webapp Pairing Testsuite S3 WebDAV FsEvents XMPP DNS

Mac OSX 10.8.2 Build 12C3006

##Please provide any additional information below.

The remote in question:

    [remote "origin"]
      url = git@git.example.com:annex-home
      fetch = +refs/heads/*:refs/remotes/origin/*
      annex-ignore = true

> belived to be [[fixed|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 1"
 date="2013-03-17T20:03:48Z"
 content="""
I was able to find two ways this could happen:

* When a removable drive was attached and the annex-ignored repository was on it.
* When the network connection went down and came back up.

In either case, it would git sync with the ignored repository. And in some cases could scan it for transfers.

I've pushed a fix for these two cases. If you can try the next autobuild, that would be helpful.

If you continue to see this happening, you should enable debug logging, and send a log after it happens.
"""]]


code:
### Please describe the problem.
git-annex assistant uses wrong key if multiple are available.  If there is only one gpg available it works without issue.

### What steps will reproduce the problem?

I tried to creating a new key through cli or assistant UI.  I also tried to select preexisting key but it always appears to choose the initial key.  Log below is creating a new key through assistant UI.

### What version of git-annex are you using? On what operating system?
git-annex version: 5.20140709-gc75193e on linux amd64 & os x 10.9.4

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

[2014-07-12 23:22:57 PDT] main: starting assistant version 5.20140709-gc75193e
[2014-07-12 23:22:57 PDT] Cronner: You should enable consistency checking to protect your data.
(Recording state in git...)
(scanning...) [2014-07-12 23:22:57 PDT] Watcher: Performing startup scan
(started...) gpg: Signature made Wed Jul  9 19:09:28 2014 PDT using DSA key ID 89C809CB
gpg: Can't check signature: public key not found

(encryption setup) (hybrid cipher with gpg key A1FE7E9BECB5C7CA) gcrypt: Development version -- Repository format MAY CHANGE
gcrypt: Repository not found: ssh://admin@nappy.local/share/backups/annex/
gcrypt: Development version -- Repository format MAY CHANGE
gcrypt: Repository not found: ssh://admin@nappy.local/share/backups/annex/
gcrypt: Setting up new repository
gcrypt: Remote ID is :id:smTV2DKvbvPIi7ftJLey
gcrypt: Encrypting to:  -R A1FE7E9BECB5C7CA
gcrypt: Requesting manifest signature
To gcrypt::ssh://admin@nappy.local/share/backups/annex/
 * [new branch]      git-annex -> git-annex
ok
[2014-07-12 23:23:42 PDT] main: Syncing with nappy.local__share_backups_annex
gcrypt: Development version -- Repository format MAY CHANGE
gcrypt: Decrypting manifest
gpg: anonymous recipient; trying secret key 6DF1BC5D ...
gpg: cannot open tty `/dev/tty': Device not configured
gcrypt: Failed to decrypt manifest!
gcrypt: Development version -- Repository format MAY CHANGE
gcrypt: Decrypting manifest
gpg: anonymous recipient; trying secret key 6DF1BC5D ...
gpg: cannot open tty `/dev/tty': Device not configured
gcrypt: Failed to decrypt manifest!

# End of transcript or log.
"""]]

[[!tag forwarded]]
[[!meta title="using git-remote-gcrypt with multiple keys causes unncessary password prompts for keys that did not encrypt the repository"]]

> [[fixed|done]]; upgrade git-remote-gcrypt and `git config gcrypt.publish-participants true`
> and after the next push to the repo, it'll stop guessing at keys, and use
> the right one. git-annex will do this by default for new gcrypt
> repositories. --[[Joey]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 1"
 date="2014-07-14T20:54:32Z"
 content="""
AFAICs, this is a known bug in git-remote-gcrypt: <https://github.com/blake2-ppc/git-remote-gcrypt/issues/9>

In your log, it encrypts to A1FE7E9BECB5C7CA. I think this is the new key that git-annex generated. The problem comes when trying to decrypt the manifest, then it tries every gpg key until it finds one that works.
"""]]


code:
[[!comment format=mdwn
 username="lex"
 ip="17.199.80.21"
 subject="comment 2"
 date="2014-07-14T22:21:32Z"
 content="""
ah.  It's bailing due to the first key requiring passphrase, and since it aborts there it never tries the second so the whole thing fails.

okay thanks for the link.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 3"
 date="2014-07-15T20:42:15Z"
 content="""
So let's think about whether gcrypt using gpg -R makes sense from the git-annex perspective.

Without -R, an attacker who can look at the remote, encrypted git repository can easily see the gpg keys of participants. This could allow them to perform other, more targeted attacks to get at the unencrypted repository.

If the user is using eg, github to store the gcrypt repo, an attacker can easily find out who owns the repo anyway, so they know who to attack then, even when -R is used. The -R can still prevent them from finding out when it's encrypted to additional users than the owner.

As far as the assistant goes, it only ever sets up a gcrypt repo with one participant. Using either an existing gpg key of the user, or generating a new one (which doesn't even have their name on it). Adding more participants to a gcrypt repo is tricky, and the assistant doesn't currently support it. But I'd like to have it support setting up multiple participants eventually. So the current limitations of the assistant are not a sufficient reason to avoid using -R.

So, it seems to come down to the question of whether it's a reasonable goal for git-annex, when used with gcrypt, to hide the identities of people who use a repository. And whether such a goal is worth the security/usability tradeoff of the user getting gpg passphrase prompts for other keys.

Hmm, when git-annex encrypts files to send them to a special remote, if it's using encryption=pubkey, it does not use -R. So an attacker can get the list of participants that way. Although the more common approach is for git-annex to encrypt using encryption=shared/hybrid, which uses a symmetric cipher, which avoids that problem. There are plenty of other things that git-annex can do that would leak identity.

It kind of seems to me that if you want to prevent anyone learning who uses your repository, you are going to need to be very cautious (using tor etc) and the  most git-annex can do is be open about how it works and avoid adding any obstacles. Looking at it this way, such a user, or group of users, would be well-served by using gpg keys that don't have their names on them...
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 4"
 date="2014-07-15T21:06:23Z"
 content="""
@lex, it shouldn't fail due to passphrase if you have gpg-agent set up. You'll only get some unncessary passphrase prompts.
"""]]


code:
a special remote (encrypted rsync) that got copied to long ago (not sure when, there are old files that already have sizes in their unencrypted file names) seems to use the aa/bb/GPGHMACSHA1-- format instead of aaa/bbb/GPGHMACSHA1-. ``git annex fsck`` over such files produces very irritating output:

<code>
fsck L1100423.JPG (gpg) (checking …remote…...)
rsync: change_dir "…somewhere…/0a0/8cd/GPGHMACSHA1--91234b770b34eeff811d09c97ce94bb2398b3d72" failed: No such file or directory (2)

sent 8 bytes  received 12 bytes  40.00 bytes/sec
total size is 0  speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1536) [Receiver=3.0.9]

  rsync failed -- run git annex again to resume file transfer

GPGHMACSHA1--91234b770b34eeff811d09c97ce94bb2398b3d72
     3922730 100%  623.81kB/s    0:00:06 (xfer#1, to-check=0/1)

sent 30 bytes  received 3923328 bytes  523114.40 bytes/sec
total size is 3922730  speedup is 1.00
(checksum...) ok
</code>

(observed with debian's git-annex 3.20121017).

while this does output an "ok" at th end and a zero exit status, having such messages in an fsck is highly irritating.

i see two ways to enhance the situation:

* silence the "not found" error when the file is found in another location
* a way to rename the files in the remote (i guess the aaa/bbb part can be derived from the file name; in that case, that could even be done w/o network interaction).

[[!tag confirmed]]


code:
[[!comment format=mdwn
 username="http://christian.amsuess.com/chrysn"
 nickname="chrysn"
 subject="also affects `git annex get`"
 date="2012-11-12T23:11:36Z"
 content="""
the same problem also shows up with `git annex get`:

    get …filename… (from prometheus...)
    rsync: change_dir \"/home/shared/photos/encrypted_storage/63e/50b/GPGHMACSHA1--b83e8aaf05918ae2fc81652368f9d4068f938625\" failed: No such file or directory (2)

    sent 8 bytes  received 12 bytes  8.00 bytes/sec
    total size is 0  speedup is 0.00
    rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1536) [Receiver=3.0.9]

      rsync failed -- run git annex again to resume file transfer

    GPGHMACSHA1--b83e8aaf05918ae2fc81652368f9d4068f938625
          513214 100%   95.68kB/s    0:00:05 (xfer#1, to-check=0/1)

    sent 30 bytes  received 513396 bytes  44645.74 bytes/sec
    total size is 513214  speedup is 1.00
    ok

again, it says \"ok\", but the \"no such file or directory\" / \"rsync failed\" is visually more prominent.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.252.11.120"
 subject="comment 2"
 date="2012-11-13T17:27:31Z"
 content="""
So, I don't know how to suppress this message without causing worse problems, like suppressing real error messages, and even password prompts.

"""]]


code:
[[!comment format=mdwn
 username="http://christian.amsuess.com/chrysn"
 nickname="chrysn"
 subject="comment 3"
 date="2012-11-14T07:31:11Z"
 content="""
how about renaming the stored files, them? if you give me a pointer on how the directory names are generated, i can write a script that does the migration (some hash of the file name?). i suppose that's just a relic from another naming scheme, isn't it?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.252.11.120"
 subject="comment 4"
 date="2012-11-14T17:31:38Z"
 content="""
The new hash directory tree is generated in a simple to explain way. Take the md5sum of the key and the first 3 characters are the first directory, and the next 3 characters are the second directory.

The old hash directory tree is rather harder to explain. It takes the md5sum of the key, but rather than a string, represents it as 4 32bit words. Only the first word is used. It is converted into a string by the same mechanism that would be used to encode a normal md5sum value into a string, but where that would normally encode the bits using the 16 characters 0-9a-f, this instead uses the 32 characters \"0123456789zqjxkmvwgpfZQJXKMVWGPF\". The first 2 letters of the resulting string are the first directory, and the second 2 are the second directory.

There's probably a 1:1 mapping between this special md5 encoding an a regular md5 encoding. But it's certainly easier just to use the existing Haskell implementation of the hash. The following program, which needs to be built inside a git-annex source tree, reads keys on stdin, and outputs their old hash directory tree values, and their new values on stdout.

import Locations
import Types.Key
import Utility.Misc

main = interact $ \s -> case file2key $ firstLine s of
        Nothing -> \"bad key\"
        Just k -> hashDirMixed k ++ \" \" ++ hashDirLower k ++ \"\n\"

joey@gnu:~/src/git-annex>ghc --make convert.hs
joey@gnu:~/src/git-annex>echo WORM--foo | ./ convert
jq/8w/ 2b1/ba3/

"""]]


code:
[[!comment format=mdwn
 username="http://christian.amsuess.com/chrysn"
 nickname="chrysn"
 subject="single-line migration"
 date="2012-11-22T03:14:45Z"
 content="""
i've successfully applied this monster to migrate my repository (as always with such expressions, use it only if you know what it does, and have a backup):

    find . -path './??/??/*' -type d \( -exec python -c 'import sys, hashlib, os; hash = hashlib.md5(sys.argv[1][8:]).hexdigest(); h1 = hash[:3]; h2 = hash[3:6]; os.mkdir(h1) if not os.path.exists(h1) else None; os.mkdir(h1+\"/\"+h2) if not os.path.exists(h1+\"/\"+h2) else None; os.rename(sys.argv[1], h1+\"/\"+h2+\"/\"+sys.argv[1][8:])' '{}' ';' -o -print \) -prune

when executed in an encrypted git annex object directory, it takes all two-letter directories, executes a python expression on them (in case of failure, printing the file name it failed on), and doesn't continue searching there (-prune avoids error messages about moved-away directories).

the python expression itself generates the hash described above, generates the required directories (put awkwardly in an `a if b else c` expression to avoid ifs (which wouldn't fit in a single line) and because python still doesn't have a proper mkdir-p function), and moves the found object there. (nb: using the system's `mkdir -p` would trigger [[another bug|bugs/using_old_remote_format_generates_irritating_output]]).
"""]]


code:
### Please describe the problem.

I started the webapp on a pre-existing repo today, and it started adding all sorts of files which I didn't manually add, which is a little bit surprising, but "okay".

The problem is that it would loop over this one file over and over again. Adding it on the commandline yields a proper error, but this doesn't seem to properly propagate to the frontend.

### What steps will reproduce the problem?

Unclear.

### What version of git-annex are you using? On what operating system?

4.20131105-g8efdc1a

### Please provide any additional information below.

Here's the daemon.log:

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

add clips/A31-05-12_21.22.amr (checksum...) [2013-11-06 10:21:10 EST] Committer: Committing changes to git
[2013-11-06 10:21:11 EST] Committer: Adding A31-05-12_21.22.amr
(Recording state in git...)
(Recording state in git...)
add clips/A31-05-12_21.22.amr (checksum...) [2013-11-06 10:21:11 EST] Committer: Committing changes to git
[2013-11-06 10:21:12 EST] Committer: Adding A31-05-12_21.22.amr
(Recording state in git...)
(Recording state in git...)
add clips/A31-05-12_21.22.amr (checksum...) [2013-11-06 10:21:12 EST] Committer: Committing changes to git
[2013-11-06 10:21:13 EST] Committer: Adding A31-05-12_21.22.amr
(Recording state in git...)
add clips/A31-05-12_21.22.amr (checksum...) [2013-11-06 10:21:13 EST] Committer: Committing changes to git
[2013-11-06 10:21:14 EST] Committer: Adding A31-05-12_21.22.amr
(Recording state in git...)
(Recording state in git...)
add clips/A31-05-12_21.22.amr (checksum...) [2013-11-06 10:21:14 EST] Committer: Committing changes to git
# End of transcript or log.
"""]]

And here's the attempt on the commandline, which is way more informative:

[[!format sh """
anarcat@marcos:video$ git annex add .
add clips/96_257.mp3 ok
add clips/A31-05-12_21.22.amr (checksum...)
git-annex: /srv/video/.git/annex/objects/z3/96/SHA256E-s260070--46f0d657cdd79032d431e2aebb04b63685ab26e7d00963036e0b64c9f86998f8.22.amr/SHA256E-s260070--46f0d657cdd79032d431e2aebb04b63685ab26e7d00963036e0b64c9f86998f8.22.amr: setFileMode: permission denied (Operation not permitted)
failed
"""]]

Turns out that file is owned by root, so it's an expected failure. But the webapp should simply skip this file, not loop over it.

The workaround is obviously to give this file to the proper user. --[[anarcat]]

> [[fixed|done]] --[[Joey]]


code:
### Please describe the problem.
I could not find a way to abort the addition of a new remote repository.

### What steps will reproduce the problem?
- start adding a remote repository (unencrypted, with git-annex installed);
- forget to create the folder on the remote host;
- navigate away from the repository page;
- the dashboard says the repository is partially set-up, and the only thing one can do is look at the log (which says the folder is missing).

I was able to solve it by creating another repository with the exact same data.

### What version of git-annex are you using? On what operating system?

Version: 4.20131002-gf25991c on OS X 10.8.5

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

# End of transcript or log.
"""]]


code:
### Please describe the problem.
The webapp doesn't start.  There is no feedback as to what is wrong when starting the OS X app.  Makes for a very confusing behaviour.

When starting git-annex-webapp directly, it says:
git-annex: no git repository in /Users/lhunath/annex

### What steps will reproduce the problem?
Create a ~/annex that is not a git repo.

### What version of git-annex are you using? On what operating system?
git-annex version: 5.20140517-g0aed6d9
build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV FsEvents XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier ddar hook external
local repository version: unknown
supported repository version: 5
upgrade supported from repository versions: 0 1 2 4

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
$ /Applications/git-annex.app/Contents/MacOS/git-annex-webapp
git-annex: no git repository in /Users/lhunath/annex
# End of transcript or log.
"""]]

> [[fixed|done]] --[[Joey]]


code:
### Please describe the problem.

Starting the webapp on a repository that was hastily created by copying an existing one with an older version yields an undecipherable error.

Rather minor.

### What steps will reproduce the problem?

1. Install git-annex from git
2. make a repo
3. copy it over to an external hard drive
4. connect that drive to a wheezy box running git-annex from backports
5. add the external hard drive to the webapp as a new repo
6. boom

I expected git-annex to tell me:

    git-annex: Repository version 5 is not supported. Upgrade git-annex.

Instead, it popped a red box saying a scary "Internal server error". I couldn't read the daemon logs either.

### What version of git-annex are you using? On what operating system?

original version is:

    git-annex version: 5.20131109-gf2cb5b9

the failing version is running the one from wheezy backports.

### Please provide any additional information below.

screenshot coming up.

> [[fixed|done]] --[[Joey]]


code:
### Please describe the problem.

My repo contains more than 1000 files, many are unicode filenames. Now that [this bug](http://git-annex.branchable.com/bugs/fails_to_get_content_from_bare_repo_on_windows/) is resolved, almost all files are fetched, except one file named '移动硬盘 1T Buffalo USB3.0 白色.rtf'.

### What steps will reproduce the problem?

I reproduced this problem by creating a repo containing only this file with no content. If anyone wants to reproduce this, just create an empty file by copy and paste the above filename in the quote.

I created the repo on mac, synced with a usb drive, then on windows machine try to sync it back.

* When run git-annex webapp in the newly created windows repo, after the usb drive is added as a remote, an error popups up: http://imgur.com/5ZfIeGQ  although the remote is added successfully (http://imgur.com/04O8kaC)
* On the command line, git annex sync runs successfully, but git annex get . failed: http://imgur.com/bIVrbe2
* The file is there (http://imgur.com/URGwWWt) with correct filename, just that the content is not there.

### What version of git-annex are you using? On what operating system?

Latest release of git-annex on both mac and windows. The initial repo on mac is indirect repo, the one on usb drive is a bare repo, the one on windows is direct repo.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.203"
 subject="comment 1"
 date="2014-06-19T22:39:46Z"
 content="""
I don't quite reproduce this on windows. Instead of the file showing up and `git annex get` not getting it, the file never shows up at all. It seems that the direct mode merge code fails to create the file in the work tree. This also means that another sync will commit the deletion of the file.

Based on the screenshot, I do have the identical filename checked into git, which git ls-files etc represents as \"\347\247\273\345\212\250\347\241\254\347\233\230 1T Buffalo USB3.0 \347\231\275\350\211\262.rtf\"

Anyway, the root cause is probably the same. See [[todo/windows_support]] for a discussion of several issues with filename encodings and windows.
"""]]


code:
### Please describe the problem.
audio file won't download when it's sole source is the internet

### What steps will reproduce the problem?
on my laptop git annex addurl --fast http://path/to/file.mp3 in folder phone/
sync to homeserver
sync to phone
phone is set to prefer phone/* and present
after the phone receives the sync from homeserver, it looks to start the download and fails.
https://www.dropbox.com/s/ozcwub2mzo40z4y/2014-01-12%2012.36.01.png -> says wget doesn't recognize option --clobber

https://www.dropbox.com/s/x1jhb6rcp17c7t8/2014-01-12%2013.13.56.png -> git annex get . -d output

### What version of git-annex are you using? On what operating system?
android is version 5.20131230-g4aa88d8 for android 4.3+

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

# End of transcript or log.
"""]]

> [[fixed|done]] --[[Joey]]


code:
### Please describe the problem.
I've a undirect repository that I sync with a direct one (that is on a
crippled filesystem). My repository contain tex file that are directly
in git, and binary file that are annexed.

If I create or modify an standard git files in my undirect repository,
then sync it with the direct repository, the new file is not created
there, or the modified file is deleted there.

### What steps will reproduce the problem?

    $ cd direct
    $ echo foo > bar.tex
    $ git commit bar.tex -m "new tex file"
    $ git annex sync
    $ cd /media/direct-repo
    $ git annex sync
    $ ls bar.tex

There the bar.tex file don't exist, and the next `git annex sync` will
commit the change.

The problem exist for some time (I first believed I add a problem with
my usb stick), but now that direct repository are treated somewhat as
bare repository, I don't know of an easy fix.

### What version of git-annex are you using? On what operating system?

On Debian, since at least git annex 5.20131120 I believe.

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
mkdir direct
cd direct
git init && git annex init "direct"
git annex direct
cd ..
git clone direct indirect
cd indirect
git annex init "indirect"
echo bla > test.tex
git add test.tex
git commit -m "test"
git annex sync
cd ../direct
git annex sync
ls
# End of transcript or log.
"""]]

> [[fixed|done]] --[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnm19dBCRphmtjXfopm_NpvnRwz-qIJ2Tw"
 nickname="Remi"
 subject="Checking out the file"
 date="2013-12-03T10:29:40Z"
 content="""
If I run `GIT_WORK_TREE=/media/direct-repo git checkout bar.tex` I have my file back.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmLdL4zLty4fsfaIXNtPMS5RXs4y8OAk0o"
 nickname="Marc"
 subject="Similar problem"
 date="2013-12-08T12:30:12Z"
 content="""
I've described the same (or similar) problem here:
http://git-annex.branchable.com/tips/replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant/#comment-015d93573c7c34d10f3815a22bdc13c0
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnpmMRq6lo2NBrlmoxZQljaU_IaWsWB7qw"
 nickname="Lukas"
 subject="direct mode and vanishing files"
 date="2013-12-10T12:40:51Z"
 content="""
Hi

I have the same problem with git-annex 4.20131106 on debian jessie (powerpc) in direct mode and 4.20131101 on mac osx 10.8.5 (self compiled from haskell) in indirect mode.
Files directly added to git on the mac (indirect mode) are not created in the direct repository and are removed in the indirect repo after the next sync roundtrip.

Thanks a lot for all your efforts and greetings, Lukas
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="comment 4"
 date="2013-12-12T19:56:50Z"
 content="""
I have fixed this bug. I apologise for the trouble.

I did not try to make it clean up, so I recommend doing the checkout as shown above, or doing a `git revert` to get the file added back, if you experienced this bug.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmLdL4zLty4fsfaIXNtPMS5RXs4y8OAk0o"
 nickname="Marc"
 subject="comment 5"
 date="2013-12-13T15:43:46Z"
 content="""
That's great, Joey, thanks a lot!
"""]]


code:
`git annex whereis` claims that I have zero copies of two particular files. However, it is wrong; both files are present:

[[!format txt """
[jkt@vorvan fotky]$ git annex whereis '2011-08-13 Svatba Anička Fellnerová a výlet s Julií/IMG_4301'*
(merging synced/git-annex into git-annex...)
whereis 2011-08-13 Svatba Anička Fellnerová a výlet s Julií/IMG_4301.CR2 (0 copies) failed
whereis 2011-08-13 Svatba Anička Fellnerová a výlet s Julií/IMG_4301.JPG (0 copies) failed
git-annex: whereis: 2 failed
[jkt@vorvan fotky]$ ll 2011-08-13\ Svatba\ Anička\ Fellnerová\ a\ výlet\ s\ Julií/IMG_4301*
lrwxrwxrwx. 1 jkt jkt 331 Aug 13  2011 2011-08-13 Svatba Anička Fellnerová a výlet s Julií/IMG_4301.CR2 -> ../.git/annex/objects/mg/qq/SHA512E-s19424303--9bdd6917fe85a450b50233a444c1814d407d41b550b66661272728aaf139576cc7c09de158f47a33ff263d7d25253cec193d0e44ed5fb4a1581c28a9a05e75e1.CR2/SHA512E-s19424303--9bdd6917fe85a450b50233a444c1814d407d41b550b66661272728aaf139576cc7c09de158f47a33ff263d7d25253cec193d0e44ed5fb4a1581c28a9a05e75e1.CR2
lrwxrwxrwx. 1 jkt jkt 329 Aug 13  2011 2011-08-13 Svatba Anička Fellnerová a výlet s Julií/IMG_4301.JPG -> ../.git/annex/objects/zG/xq/SHA512E-s4744850--da9b1e19f0ecdf08d2f5564232a9c7d4b5ec9fbc9c67033121e5c90468e09e5573ddac42469cfd51fb0d8328dff925830e2d753022365df6d92b115b77831228.JPG/SHA512E-s4744850--da9b1e19f0ecdf08d2f5564232a9c7d4b5ec9fbc9c67033121e5c90468e09e5573ddac42469cfd51fb0d8328dff925830e2d753022365df6d92b115b77831228.JPG
"""]]

Looking at another repo, the situation is the same:

[[!format txt """
jkt@svist /mnt/storage5/fotky[master] $ git annex whereis '2011-08-13 Svatba Anička Fellnerová a výlet s Julií/IMG_4301'*
whereis 2011-08-13 Svatba Anička Fellnerová a výlet s Julií/IMG_4301.CR2 (0 copies) failed
whereis 2011-08-13 Svatba Anička Fellnerová a výlet s Julií/IMG_4301.JPG (0 copies) failed
git-annex: whereis: 2 failed
jkt@svist /mnt/storage5/fotky[master] $ ll '2011-08-13 Svatba Anička Fellnerová a výlet s Julií/IMG_4301'*
lrwxrwxrwx 1 jkt jkt 331 Mar  3 02:08 2011-08-13 Svatba Anička Fellnerová a výlet s Julií/IMG_4301.CR2 -> ../.git/annex/objects/mg/qq/SHA512E-s19424303--9bdd6917fe85a450b50233a444c1814d407d41b550b66661272728aaf139576cc7c09de158f47a33ff263d7d25253cec193d0e44ed5fb4a1581c28a9a05e75e1.CR2/SHA512E-s19424303--9bdd6917fe85a450b50233a444c1814d407d41b550b66661272728aaf139576cc7c09de158f47a33ff263d7d25253cec193d0e44ed5fb4a1581c28a9a05e75e1.CR2
lrwxrwxrwx 1 jkt jkt 329 Mar  3 02:08 2011-08-13 Svatba Anička Fellnerová a výlet s Julií/IMG_4301.JPG -> ../.git/annex/objects/zG/xq/SHA512E-s4744850--da9b1e19f0ecdf08d2f5564232a9c7d4b5ec9fbc9c67033121e5c90468e09e5573ddac42469cfd51fb0d8328dff925830e2d753022365df6d92b115b77831228.JPG/SHA512E-s4744850--da9b1e19f0ecdf08d2f5564232a9c7d4b5ec9fbc9c67033121e5c90468e09e5573ddac42469cfd51fb0d8328dff925830e2d753022365df6d92b115b77831228.JPG
"""]]

The directory names are valid UTF-8. These are very common on my machine and there is a ton of directories with these funny names here -- all working without any real trouble.

As far as I know, the file which the links point to is absolutely correct and not corrupted. Looking at the files in the directory chronologically, it also appears that the symlinks point to a correct file.

[[!tag moreinfo]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.146"
 subject="comment 1"
 date="2014-03-05T03:14:09Z"
 content="""
Have you tried running `git annex fsck`?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.102"
 subject="ping?"
 date="2014-03-19T20:24:23Z"
 content="""
Have you tried running git-annex fsck?
"""]]


code:
### Please describe the problem.

`git annex whereis` says that there are no copies of any of the files annexed in repositories running in direct mode.

This is the error received:

    $ git annex whereis
    whereis fileA (0 copies) failed
    whereis fileB (0 copies) failed
    git-annex: whereis: 2 failed

### What steps will reproduce the problem?

The following script (available at <https://gist.github.com/gioele/dde462df89edfe17c5e3>) will reproduce this problem.

[[!format sh """
#!/bin/sh -x

set -e ; set -u
export LC_ALL=C

direct=true # set to false to make the problem disappear

h=${h:-localhost}
dr="/tmp/annex"

sync='sync -c annex.alwayscommit=true'

chmod a+rwx -R pc1 pc2 || true
rm -Rf pc1 pc2

# create central git repo
ssh $h "chmod a+rwx -R ${dr}/Docs.git" || true
ssh $h "rm -Rf ${dr}/Docs.git"
ssh $h "mkdir -p ${dr}/Docs.git"
ssh $h "cd ${dr}/Docs.git ; git init --bare"

d=$(pwd)

# populate repo in PC1
mkdir -p pc1/Docs
cd pc1/Docs
echo AAA > fileA
echo BBB > fileB

git init
git remote add origin $h:$dr/Docs.git
git fetch --all

# simulate a host without git-annex
git config remote.origin.annex-ignore true

git annex init "pc1"
git annex info

$direct && git annex direct

git annex add .
git annex $sync origin

# re-create repo on PC2
cd $d
mkdir -p pc2
cd pc2
git clone $h:$dr/Docs.git
cd Docs

git config remote.origin.annex-ignore true

git annex init "pc2"
git annex info

git annex whereis || true
echo "I was expecting location info to be available after info (press Enter)" ; read enter

git annex $sync origin

git annex whereis || true
echo "Why isn't location info available even after sync? (press Enter)"
"""]]

### What version of git-annex are you using? On what operating system?

git-annex version: 5.20140708-g42df533


code:
What steps will reproduce the problem?

    ...:/tmp$ mkdir repro
    ...:/tmp$ cd repro/
    ...:/tmp/repro$ git init
    Initialized empty Git repository in /tmp/repro/.git/
    ...:/tmp/repro$ git annex init test
    init test ok
    ...:/tmp/repro$ echo "A" > a.txt
    ...:/tmp/repro$ git annex add a.txt
    add a.txt (checksum...) ok
    (Recording state in git...)
    ...:/tmp/repro$ git commit -m "add file"
    [master (root-commit) bf53ce2] add file
     1 file changed, 1 insertion(+)
     create mode 120000 a.txt
    ...:/tmp/repro$ git annex whereis a.txt
    whereis a.txt (1 copy)
      	5c028c6a-2c5e-11e2-bb9c-17bd7ce81377 -- here (test)
    ok
    ...:/tmp/repro$ git annex unlock a.txt
    unlock a.txt (copying...) ok
    ...:/tmp/repro$ git annex whereis a.txt

What is the expected output? What do you see instead?

    I'd expect that whereis executed on an unlocked file would behave like whereis executed on a locked file.

What version of git-annex are you using? On what operating system?

    $ cat /etc/issue
    Ubuntu 12.04.1 LTS \n \l

    $ git-annex version
    git-annex version: 3.20120406
    local repository version: 3
    default repository version: 3
    supported repository versions: 3
    upgrade supported from repository versions: 0 1 2

    $ uname -a
    Linux ... 3.2.0-31-generic #50-Ubuntu SMP Fri Sep 7 16:16:45 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

Please provide any additional information below.

[[!tag confirmed]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.102"
 subject="comment 1"
 date="2014-03-19T20:52:08Z"
 content="""
The reason this doesn't work is that, in indirect mode, git-annex looks at the current state of the symlink in the work tree to know what key is associated with a file. And an unlocked file has no symlink.

Direct mode avoids this problem, but at the expense of being less flexible and well, doing more work.
"""]]


code:
### Please describe the problem.
When installing the windows aplha dated 2013-06-21 12:17 it gives an error:

Error opening file for writing:
c:\program files (x86)\Git\cmd\git-annex.exe
Abort,retry or ignore?

If you run as administrator it works.

### What steps will reproduce the problem?

Download the alpha installer and run as normal user.

### What version of git-annex are you using? On what operating system?
Windows 7 64 bit

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

# End of transcript or log.
"""]]

[[!meta title="windows installer fails unless run as admin user"]]

> [[dup|done]] of
> [[git-annex_does_not_install_on_windows_without_admin_rights]] --[[Joey]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnI6EUnvCSB6JG3KsjCDpBIBWYniaZkhXs"
 nickname="Jason"
 subject="expected behavior"
 date="2013-06-27T02:44:37Z"
 content="""
I expect it should automatically ask for privileged escalation rather than fail.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnrFnHRRNUQBB5RCDaIwhVmCcxQp8_yiYw"
 nickname="Oliver"
 subject="comment 2"
 date="2013-06-28T08:37:39Z"
 content="""
Indeed, but the windows installer (whether built by hand or from the prebuilts) does not. Might have a poke at it when I'm back at my machine again - should be a fairly easy fix.
"""]]


code:
### Please describe the problem.

[[This|design/assistant/blog/day_314__quvi/]] says that I can download youtube videos with git-annex, but I can't - it downloads the HTML of the page instead!

### What steps will reproduce the problem?

    git annex addurl http://www.youtube.com/watch?v=BTzNLhxPzjo

Should have downloaded a Bill Hicks video, instead it downloads the HTML.

### What version of git-annex are you using? On what operating system?

[[!format txt """
git-annex version: 4.20130912-ga1faca3
build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP DNS Feeds Quvi
local repository version: 3
default repository version: 3
supported repository versions: 3 4
upgrade supported from repository versions: 0 1 2
"""]]

Note that quvi is still capable of fetching that video, no problem there.

### Please provide any additional information below.

[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
anarcat@marcos:clips$ git annex addurl http://www.youtube.com/watch?v=BTzNLhxPzjo
addurl www.youtube.com_watch_v=BTzNLhxPzjo ok
anarcat@marcos:clips$ file www.youtube.com_watch_v\=BTzNLhxPzjo
www.youtube.com_watch_v=BTzNLhxPzjo: broken symbolic link to `.git/annex/objects/qZ/q1/SHA256E-s123162--96fe8634aaf035373077473ac2b166c5ad22bfe099112c414eee9fe753f4ce2b/SHA256E-s123162--96fe8634aaf035373077473ac2b166c5ad22bfe099112c414eee9fe753f4ce2b'
anarcat@marcos:clips$ file www.youtube.com_watch_v\=BTzNLhxPzjo ^C
anarcat@marcos:clips$ file .git/annex/objects/qZ/q1/SHA256E-s123162--96fe8634aaf035373077473ac2b166c5ad22bfe099112c414eee9fe753f4ce2b/SHA256E-s123162--96fe8634aaf035373077473ac2b166c5ad22bfe099112c414eee9fe753f4ce2b
.git/annex/objects/qZ/q1/SHA256E-s123162--96fe8634aaf035373077473ac2b166c5ad22bfe099112c414eee9fe753f4ce2b/SHA256E-s123162--96fe8634aaf035373077473ac2b166c5ad22bfe099112c414eee9fe753f4ce2b: ERROR: cannot open `.git/annex/objects/qZ/q1/SHA256E-s123162--96fe8634aaf035373077473ac2b166c5ad22bfe099112c414eee9fe753f4ce2b/SHA256E-s123162--96fe8634aaf035373077473ac2b166c5ad22bfe099112c414eee9fe753f4ce2b' (No such file or directory)
anarcat@marcos:clips$ git annex get www.youtube.com_watch_v\=BTzNLhxPzjo
get www.youtube.com_watch_v=BTzNLhxPzjo (from web...)
--2013-09-14 17:24:23--  http://www.youtube.com/watch?v=BTzNLhxPzjo
Résolution de www.youtube.com (www.youtube.com)... 173.194.43.96, 173.194.43.97, 173.194.43.98, ...
Connexion vers www.youtube.com (www.youtube.com)|173.194.43.96|:80...connecté.
requête HTTP transmise, en attente de la réponse...200 OK
Longueur: non spécifié [text/html]
Sauvegarde en : «/srv/video/clips/.git/annex/tmp/SHA256E-s123162--96fe8634aaf035373077473ac2b166c5ad22bfe099112c414eee9fe753f4ce2b»

    [  <=>                                                                                ] 123 245      346K/s   ds 0,3s

2013-09-14 17:24:23 (346 KB/s) - «/srv/video/clips/.git/annex/tmp/SHA256E-s123162--96fe8634aaf035373077473ac2b166c5ad22bfe099112c414eee9fe753f4ce2b» sauvegardé [123245]

ok
(Recording state in git...)
anarcat@marcos:clips$ file www.youtube.com_watch_v\=BTzNLhxPzjo
www.youtube.com_watch_v=BTzNLhxPzjo: symbolic link to `.git/annex/objects/qZ/q1/SHA256E-s123162--96fe8634aaf035373077473ac2b166c5ad22bfe099112c414eee9fe753f4ce2b/SHA256E-s123162--96fe8634aaf035373077473ac2b166c5ad22bfe099112c414eee9fe753f4ce2b'
anarcat@marcos:clips$ file .git/annex/objects/qZ/q1/SHA256E-s123162--96fe8634aaf035373077473ac2b166c5ad22bfe099112c414eee9fe753f4ce2b/SHA256E-s123162--96fe8634aaf035373077473ac2b166c5ad22bfe099112c414eee9fe753f4ce2b
.git/annex/objects/qZ/q1/SHA256E-s123162--96fe8634aaf035373077473ac2b166c5ad22bfe099112c414eee9fe753f4ce2b/SHA256E-s123162--96fe8634aaf035373077473ac2b166c5ad22bfe099112c414eee9fe753f4ce2b: HTML document, UTF-8 Unicode text, with very long lines
# End of transcript or log.
"""]]

### Workaround

Use the `quvi` prefix.

> Typo introduced in last release. [[fixed|done]] --[[Joey]]


code:
### Please describe the problem.

I'm downloading a video with 'git annex addurl' from youtube. The webapp shows the transfer, but when I click the 'web' link it takes me to a page that says "internal server error" and "Prelude.undefined"

### What steps will reproduce the problem?

1. start webapp
2. download youtube video
3. click 'web' link on transfer in git-annex

### What version of git-annex are you using? On what operating system?

* Version: 5.20140717
* OS: Debian sid

### Please provide any additional information below.

Not much in the logs, I see this:

[[!format sh """

[2014-07-25 08:40:14 BST] TransferWatcher: transfer starting: Download UUID "00000000-0000-0000-0000-000000000001" URL--quvi:https://www.youtube.com/watch,63v,61Z8_8jNLsZms Nothing
[2014-07-25 08:40:14 BST] TransferWatcher: transfer starting: Download UUID "00000000-0000-0000-0000-000000000001" Chase_Adam_at_Startup_School_NY_2014.mp4 Nothing

"""]]


code:
[[!meta title="git-annex autobuild overview"]]

[[!sidebar content="""
# last uploaded build-versions

Linux i386

Linux amd64

Linux armel

Android

OSX Mavericks

Windows

"""]]

# build logs

Linux i386

Linux amd64

Linux armel

Android

OSX Mavericks

Windows

here


code:
Some [[special_remotes]] have support for breaking large files up into
chunks that are stored on the remote.

This can be useful to work around limitations on the size of files
on the remote.

Chunking also allows for resuming interrupted downloads and uploads.

Note that git-annex has to buffer chunks in memory before they are sent to
a remote. So, using a large chunk size will make it use more memory.

To enable chunking, pass a `chunk=nnMiB` parameter to `git annex
initremote`, specifying the chunk size.

Good chunk sizes will depend on the remote, but a good starting place
is probably `1MiB`. Very large chunks are problimatic, both because
git-annex needs to buffer one chunk in memory when uploading, and because
a larger chunk will make resuming interrupted transfers less efficient.
On the other hand, when a file is split into a great many chunks,
there can be increased overhead of making many requests to the remote.

To disable chunking of a remote that was using chunking,
pass `chunk=0` to `git annex enableremote`. Any content already stored on
the remote using chunks will continue to be accessed via chunks, this
just prevents using chunks when storing new content.

To change the chunk size, pass a `chunk=nnMiB` parameter to
`git annex enableremote`. This only affects the chunk sized used when
storing new content.

See also: [[design document|design/assistant/chunks]]


code:
If you do nothing else, avoid use of partial functions from the Prelude!
`import Utility.PartialPrelude` helps avoid this by defining conflicting
functions for all the common ones. Also avoid `!!`, it's partial too.

Use tabs for indentation. The one exception to this rule are
the Hamlet format files in `templates/*`. Hamlet, infuriatingly, refuses
to allow tabs to be used for indentation.

Code should make sense with any tab stop setting, but 8 space tabs are
the default. With 8 space tabs, code should not exceed 80 characters
per line. (With larger tabs, it may of course.)

Use spaces for layout. For example, here spaces (indicated with `.`)
are used after the initial tab to make the third test line up with
the others.

		when (foo_test || bar_test ||
		......some_other_long_test)
			print "hi"

As a special Haskell-specific rule, "where" clauses are indented with two
spaces, rather than a tab. This makes them stand out from the main body
of the function, and avoids excessive indentation of the where cause content.
The definitions within the where clause should be put on separate lines,
each indented with a tab.

	main = do
		foo
		bar
		foo
	  where
		foo = ...
		bar = ...

Where clauses for instance definitions and modules tend to appear at the end
of a line, rather than on a separate line.

	module Foo (Foo, mkFoo, unFoo) where
	instance MonadBaseControl IO Annex where

When a function's type signature needs to be wrapped to another line,
it's typical to switch to displaying one parameter per line.

	foo :: Bar -> Baz -> (Bar -> Baz) -> IO Baz

	foo'
		:: Bar
		-> Baz
		-> (Bar -> Baz)
		-> IO Baz

Note that the "::" then starts its own line. It is not put on the same
line as the function name because then it would not be guaranteed to line
up with the "->" at all tab width settings. Similarly, guards are put
on their own lines:

	splat i
		| odd i = error "splat!"
		| otherwise = i

Multiline lists and record syntax are written with leading commas,
that line up with the open and close punctuation.

	list =
		[ item1
		, item2
		, item3
		]

	foo = DataStructure
		{ name = "bar"
		, address = "baz"
		}

Module imports are separated into two blocks, one for third-party modules,
and one for modules that are part of git-annex. (Additional blocks can be used
if it makes sense.)

Using tabs for indentation makes use of `let .. in` particularly tricky.
There's no really good way to bind multiple names in a let clause with
tab indentation. Instead, a where clause is typically used. To bind a single
name in a let clause, this is sometimes used:

	foo = let x = 42
		in x + (x-1) + x

-----

If you feel that this coding style leads to excessive amounts of horizontal
or vertical whitespace around your code, making it hard to fit enough of it
on the screen, consider finding a better abstraction, so the code that
does fit on the screen is easily understandable. ;)


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnq-RfkVpFN15SWvQ2lpSGAi0XpNQuLxKM"
 nickname="Yuval"
 subject="What about safe?"
 date="2014-07-06T10:45:59Z"
 content="""
https://hackage.haskell.org/package/safe

> A library wrapping Prelude/Data.List functions that can throw exceptions, such as head and !!. Each unsafe function has up to four variants, e.g. with tail:
>
>     tail :: [a] -> [a], raises an error on tail [].

>     tailMay :: [a] -> Maybe [a], turns errors into Nothing.

>     tailDef :: [a] -> [a] -> [a], takes a default to return on errors.

>     tailNote :: String -> [a] -> [a], takes an extra argument which supplements the error message.

>     tailSafe :: [a] -> [a], returns some sensible default if possible, [] in the case of tail.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.2"
 subject="comment 2"
 date="2014-07-10T20:24:03Z"
 content="""
safe does not prevent using the unsafe prelude functions. Utility.PartialPrelude does, and provides a few safe wrappers like lastMaybe.
"""]]


code:
[[!sidebar content="""
[[!inline pages="comment_pending(*)" feedfile=pendingmoderation
description="comments pending moderation" show=-1]]
Comments in the [[!commentmoderation desc="moderation queue"]]:
[[!pagecount pages="comment_pending(*)"]]
"""]]

Recent comments posted to this site:
[[!inline pages="comment(*)" template="comment"]]


code:
Joey Hess <joey@kitenet.net> is the author of git-annex. If you need to
talk about something privately, email me.

The [[forum]] is the best place to discuss git-annex.

For realtime chat, use the `#git-annex` channel on irc.oftc.net.
You can also watch incoming commits there.

The [VCS-home mailing list](http://lists.madduck.net/listinfo/vcs-home)
is a good mailing list for users who want to use git-annex in the context
of managing their large personal files.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlZQcpoRmyZBlsFQ7TMMYlZZdmBG1TxMzA"
 nickname="Felipe"
 subject="translation"
 date="2013-09-10T20:19:53Z"
 content="""
I would like to translate this content of site, the tutorials, pages, etc, to portuguese. I think that can help many users in my country. How can I build a new pages here with git-annex content in portuguese?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.51"
 subject="comment 2"
 date="2013-09-12T21:08:16Z"
 content="""
I welcome translations in theory, but I do not have any infrastructure in place to support it, so that would have to be the first step.

(Also, this page is not a general-purpose discussion forum.)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlkA6XinbeOdnEDxEGQUWyjqPGh0kdMXr4"
 nickname="Blake"
 subject="Starting git-annex assistant"
 date="2013-10-02T23:57:48Z"
 content="""
Hi Joey,

I am one of your original funders for git-annex. I think you have done a great job and I am proud to say I helped fund your endeavor.

How do I start the git-annex assistant webapp via command line? I installed the program on my Fedora 19 computer via YUM and it does not come packaged with a desktop icon as you show in your demonstration video. I try running `$ git annex assistant` but I cannot access `http://127.0.0.1:34795/` as you do in the video. Also, the quickstart page (http://git-annex.branchable.com/assistant/quickstart/) seems to have outdated information as it states you can start the webapp via `$ git annex webapp`, however when I try to run this I get the following error: `git-annex: unknown command webapp`.

Regards,
Blake

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="108.236.230.124"
 subject="comment 4"
 date="2013-10-03T00:06:11Z"
 content="""
Your fedora build apparently does not include the webapp, since `git annex webapp` is the right command. Install the [[install/Linux_standalone]] build instead.

(Again this page is not a discussion forum. Next person to post here will waste my time since I will have to go configure the site to block comments here..)
"""]]


code:
Help make git-annex better!

## wiki gardening

This website is a wiki, so you can edit and improve any page.

Or, write a [[new_tip|tips]] explaining how to accomplish something with
git-annex.

## bug triage

People often file [[bugs]] on git-annex that are easily resolved by helping
them with a problem, and perhaps writing some documentation.

Many bugs are also filed without enough information
to reproduce the problem, and need to be tagged moreinfo and a comment
posted asking the submitter for details.

Joey spends a lot of time dealing with this kind of bug triage. If you can
take the time to pick a bug that is not marked as "confirmed" or "moreinfo"
from the list of [[bugs]], try to reproduce it and follow up either
confirming that the problem exists, or asking the submitter for more info,
you'll make Joey more productive!

## code contributions

[[download]] the source code and send patches!

If you know Haskell, git-annex has lots of Haskell code that
could be improved. See the [[coding_style]] and have at it.

If you don't know Haskell, git-annex has many other coding opportunities.
You could work to improve the Android port (Java etc) or improve the
Javascript and CSS of the git-annex webapp, or work on porting libraries
needed by the Windows port.

To send patches, either include the patch in a bug report (small patch)
or put up a branch in a git repository containing your changes.


code:
Annexed data is stored inside  your git repository's `.git/annex` directory.
Some [[special_remotes]] can store annexed data elsewhere.

It's important that data not get lost by an ill-considered `git annex drop`
command.  So, git-annex can be configured to try
to keep N copies of a file's content available across all repositories.
(Although [[untrusted_repositories|trust]] don't count toward this total.)

By default, N is 1; it is configured by running `git annex numcopies N`.
This default can be overridden on a per-file-type basis by the annex.numcopies
setting in `.gitattributes` files. The --numcopies switch allows
temporarily using a different value.

`git annex drop` attempts to check with other git remotes, to check that N
copies of the file exist. If enough repositories cannot be verified to have
it, it will retain the file content to avoid data loss. Note that
[[trusted_repositories|trust]] are not explicitly checked.

For example, consider three repositories: Server, Laptop, and USB. Both Server
and USB have a copy of a file, and N=1. If on Laptop, you `git annex get
$file`, this will transfer it from either Server or USB (depending on which
is available), and there are now 3 copies of the file.

Suppose you want to free up space on Laptop again, and you `git annex drop` the file
there. If USB is connected, or Server can be contacted, git-annex can check
that it still has a copy of the file, and the content is removed from
Laptop. But if USB is currently disconnected, and Server also cannot be
contacted, it can't verify that it is safe to drop the file, and will
refuse to do so.

With N=2, in order to drop the file content from Laptop, it would need access
to both USB and Server.

For more complicated requirements about which repositories contain which
content, see [[required_content]].


code:
[[!comment format=mdwn
 username="http://olivier.berger.myopenid.com/"
 nickname="obergix"
 subject="See also section on backups in walthroug"
 date="2013-08-25T05:37:04Z"
 content="""
See also [walthrough/backups](/walkthrough/backups/) for some details about copies.
"""]]


code:
git-annex's high-level design is mostly inherent in the data that it
stores in git, and alongside git. See [[internals]] for details.

Here's the other design documents we have:

[[!map pages="page(design/*) and !design/*/*"]]


code:
These are the design pages for the git-annex [[/assistant]].

Parts of the design is still being fleshed out, still many ideas
and use cases to add. Feel free to chip in with comments! --[[Joey]]

See [[roadmap]] for current plans, as this list was mostly completed.

## initial development kickstarter year overview (2012-2013)

* Month 1 "like dropbox": [[!traillink inotify]] [[!traillink syncing]]
* Month 2 "shiny webapp": [[!traillink webapp]] [[!traillink progressbars]]
* Month 3 "easy setup": [[!traillink configurators]] [[!traillink pairing]]
* Month 4 "cloud": [[!traillink cloud]] [[!traillink transfer_control]]
* Month 5 "cloud continued": [[!traillink xmpp]] [[!traillink more_cloud_providers]]
* Month 6 "9k bonus round": [[!traillink desymlink]]
* Month 7: user-driven features and polishing;
  [presentation at LCA2013](http://mirror.linux.org.au/linux.conf.au/2013/mp4/gitannex.mp4)
* Month 8: [[!traillink Android]]
* Month 9: [[screencasts|videos]] and polishing
* Month 10: bugfixing, [[Android]] webapp
* Month 11: [[!traillink Windows]] porting, finishing touches
* Month 12: [presentation at SELF2013](http://www.southeastlinuxfest.org/), finishing touches

## porting

* [[OSX]] port is in fairly good shape, but still has some room for improvement
* [[android]] port is zooming along
* [[Windows]] port is barely getting started

## not yet on the map:

* [[rate_limiting]]
* [[partial_content]]
* [[deltas]]
* [[leftovers]]
* [[other todo items|todo]]

## polls

I post [[polls]] occasionally to make decisions. You can vote!

## blog

I'm blogging about my progress in the [[devblog]] on a semi-daily basis.
Follow along!


code:
Misc OSX porting things:

* autostart the assistant on OSX, using launchd **done**
* icon to start webapp **done**
* use FSEvents to detect file changes (better than kqueue) **done**
* Use OSX's "network reachability functionality" to detect when on a network
  <http://developer.apple.com/library/mac/#documentation/Networking/Conceptual/SystemConfigFrameworks/SC_Intro/SC_Intro.html#//apple_ref/doc/uid/TP40001065>
* Switch from gpg to <https://gpgtools.org/>. According to a user,
  this is better because it can show a dialog window for password prompts.

Bugs:

[[!inline pages="tagged(design/assistant/OSX) and !link(bugs/done)" show=0 archive=yes]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlu-fdXIt_RF9ggvg4zP0yBbtjWQwHAMS4"
 nickname="Jörn"
 subject="Mount detection"
 date="2012-09-21T09:23:34Z"
 content="""
regarding the current mount polling on OSX: why not use the NSNotificationCenter for being notified on mount events on OSX?

Details see:

1. <http://stackoverflow.com/questions/12409458/detect-when-a-volume-is-mounted-on-os-x>
2. <http://stackoverflow.com/questions/6062299/how-to-add-an-observer-to-nsnotificationcenter-in-a-c-class-using-objective-c>
3. <https://developer.apple.com/library/mac/#documentation/CoreFoundation/Reference/CFNotificationCenterRef/Reference/reference.html#//apple_ref/doc/uid/20001438>
"""]]


code:
The Android port is just about usable. Still, we have some fun todo items
to improve it.

## high-priority TODO

* [[bugs/Android_app_permission_denial_on_startup]]
* S3 doesn't work (at least to Internet Archive:
  "connect: does not exist (connection refused)")
* Get app into Google Play and/or FDroid

## TODO

* Don't make app initially open terminal + webapp, but go to a page that
  allows opening the webapp or terminal.
  Possibly, switch from running inside terminal app to real standalone app.
  See <https://github.com/neurocyte/android-haskell-activity>
  and <https://github.com/neurocyte/foreign-jni>.

* I have seen an assistant thread crash with an interrupted system call
  when the device went to sleep while it was running. Auto-detect and deal with
  that somehow.
* Make git stop complaining that "warning: no threads uspport, ignoring --threads"
* git does not support http remotes. To fix, need to port libcurl and
  allow git to link to it.
* getEnvironment is broken on Android <https://github.com/neurocyte/ghc-android/issues/7>
  and a few places use it. I have some horrible workarounds in place.
* Get local pairing to work. network-multicast and network-info don't
  currently install.
* Get test suite to pass. `git clone` of a local repo fails on android
  for some reason.
* Make app autostart on boot, optionally. <http://stackoverflow.com/questions/1056570/how-to-autostart-an-android-application>
* The app should be aware of power status, and avoid expensive background
  jobs when low on battery or run flat out when plugged in.
* The app should be aware of network status, and avoid expensive data
  transfers when not on wifi. This may need to be configurable.
* glacier and local pairing are not yet enabled for Android.
* The "Files" link doesn't start a file browser. Should be possible to do
  on Android via intents, I suppose?
* Adding removable drives would work, but the android app is not in the
  appropriate group to write to them. `WRITE_MEDIA_STORAGE` permission
  needed. Added to AndroidManifest, but did not seem to be used.
  Googleing for it will find a workaround that needs a rooted device.


code:
[[!comment format=mdwn
 username="Xyem"
 ip="87.194.19.134"
 subject="comment 10"
 date="2013-05-06T12:18:39Z"
 content="""
One of the points is:

* The app should be aware of power status, and avoid expensive background jobs when low on battery or run flat out when plugged in.

This isn't good for situations where the device is plugged in and charging, but will only be so for a limited time (such as charging during a car journey) as the device would charge slower.

Could this behaviour be configurable too?
"""]]


code:
[[!comment format=mdwn
 username="https://launchpad.net/~gdr-go2"
 nickname="gdr-go2"
 subject="FAT symlinks"
 date="2012-05-28T18:12:10Z"
 content="""
It's a linux kernel so perhaps another option would be to create a big file and mount -o loop
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlhIMPGF1E0XEJKV6j6-PFzAxA1-nIlydo"
 nickname="Bernhard"
 subject="re @  gdr-go2 "
 date="2012-09-10T19:34:00Z"
 content="""
`mount` requires root and you'll have still the 4gb limit for your image by FAT. some phones (e.g. galaxy nexus) already use `ext4` for `/sdcard` though.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlhIMPGF1E0XEJKV6j6-PFzAxA1-nIlydo"
 nickname="Bernhard"
 subject="GHC and Android"
 date="2012-09-10T21:58:28Z"
 content="""
I played around a bit with GHC and Android today. It isn't really a result, but maybe useful for someone out there.

I have a Debian `chroot` environment on my Android device (howto: <http://www.saurik.com/id/10/>). In the Debian box:

    $ cat arm.hs
    main = do
      putStrLn \"Hello ARM\"
    $ ghc -static --make arm.hs
    Linking arm ...
    $ ldd arm
	libgmp.so.3 => /usr/lib/libgmp.so.3 (0x40233000)
	libm.so.6 => /lib/libm.so.6 (0x400c8000)
	libffi.so.5 => /usr/lib/libffi.so.5 (0x401b1000)
	librt.so.1 => /lib/librt.so.1 (0x40171000)
	libdl.so.2 => /lib/libdl.so.2 (0x40180000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x4018b000)
	libc.so.6 => /lib/libc.so.6 (0x40282000)
	/lib/ld-linux.so.3 (0x400a2000)
	libpthread.so.0 => /lib/libpthread.so.0 (0x4007e000)

well, that isn't really static. tell the linker to build a static binary (those are arguments to `ld`):

    $ ghc --make arm.hs -optl-static -optl-pthread
    [1 of 1] Compiling Main             ( arm.hs, arm.o )
    Linking arm ...
    $ file arm
    arm: ELF 32-bit LSB executable, ARM, version 1 (SYSV), statically linked, for GNU/Linux 2.6.18, not stripped
    $ ldd arm
    	not a dynamic executable
    $ ./arm
    Hello ARM

now, get this (quite big) binary into the normal android environment, using `adb`, `SSHDroid` or whatever:

    % cd /data/local/tmp # assuming destination of file transfer
    % ./arm
    arm: mkTextEncoding: invalid argument (Invalid argument)

looking in the source of `System.IO` it seems like an `iconv` issue. So, there's still some dynamic dependency in there... *sigh*
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.8.30"
 subject="comment 4"
 date="2012-09-11T02:12:45Z"
 content="""
Thanks Bernard, that's really massively useful. It makes sense -- statically building with libc should work, the Android kernel is still Linux after all.

To get past the iconv problem, I think all you need is part of the `locales` package from your linux system installed on the Android. Probably just a few of the data files from /usr/share/i18n/charmaps/
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlhIMPGF1E0XEJKV6j6-PFzAxA1-nIlydo"
 nickname="Bernhard"
 subject="comment 5"
 date="2012-09-11T09:34:42Z"
 content="""
`/usr/share/i18n/` does not exists on my Debian ARM system :/

however, `strace ./arm` in the debian chroot reveals that some files from `/usr/lib/gconv/` are loaded:

    [...]
    open(\"/usr/lib/gconv/UTF-32.so\", O_RDONLY) = 3
    read(3, \"\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\4\4\0\0004\0\0\0\"..., 512) = 512
    [...]

full log: <http://wien.tomnetworks.com/strace_arm>.  Unfortunately, I don't have `strace` in the android userland for comparison.

Just copying the related `gconv` files didn't work. I don't have so much time at the moment, I'll investigate further in some days or so.

At least, output using `error :: String -> a` does work :-)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 6"
 date="2012-10-30T08:44:32Z"
 content="""
Just so this does not get lost: For better or for worse, the vanilla Android devices stopped shipping with micro-SD support in 2011 (or 2010 if the Nexus S does not support them either; on sketchy GPRS so not googling around). Most higher-end Android devices ship with at least 8 GiB of on-board Flash storage, some even go up to 64 GiB.

IMHO, this would make it viable to first get git-annex working on Android without regard for FAT.

The obvious advantage is that porting should be easier and quicker.

The obvious downside is that this may mean revisiting some parts of the code later.

-- Richard
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 7"
 date="2012-10-30T08:51:17Z"
 content="""
Actually, this is something that would be ideal for a poll:

Should FAT-based Android repos be implemented:

* Immediately
* After the initial release of the Android app is working
* Before the kickstarter ends
* Not at all

Also, as another data point, the FAT-based SD card can be mounted as USB storage by any computer an Android device is connected to whereas the EXT4-based root FS can only be accessed via MTP.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnZEanlyzay_QlEAL0CWpyZcRTyN7vay8U"
 nickname="Carlo"
 subject="Terminal IDE"
 date="2012-11-21T21:46:19Z"
 content="""
I use Terminal IDE on android, which is 'sort of a shell environment'.

It has bash, git, dropbear ssh, and vim. It is a bit limited, but it does feel like a linux shell.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl7ciFZWmffuw6sLRww3CcL_F5ItsttL9w"
 nickname="Aaron"
 subject="Feature request: Events triggered on wifi SSID or connection state"
 date="2013-01-08T13:04:05Z"
 content="""
I'd like to have my phone sync a bunch of files when I'm at home and when I get to work, have them synced to my work machine, deleted off my phone and when I get home, also deleted off my home machine.

This will allow incremental backups to a local folder at home and I can then \"mule\" my files, sneaker-net style, to work with zero user intervention. It only works if events can be triggered or I can combine a triggering app with a special android intent for both ends of my trip.
"""]]


code:
This blog was updated on a semi-daily basis by Joey during the year of work
concentrating on the git-annex assistant that was funded by his kickstarter
campaign.

Post-kickstarter work will instead appear on the [[devblog]]. However,
this page's RSS feed will continue to work, so you don't have to migrate
your RSS reader.

[[!inline pages="page(devblog/*)" show=-1 feedshow=7]]

[[!inline pages="page(design/assistant/blog/*)" show=0 feeds=no]]


code:
Preferred content control is wired up to `--auto` and working for `get`,
`copy`, and `drop`. Note that `drop --from remote --auto` drops files that
the remote's preferred content settings indicate it doesn't want;
likewise `copy --to remote --auto` sends content that the remote does want.

Also implemented `smallerthan`, `largerthan`, and `ingroup` limits,
which should be everything needed for the scenarios described in
[[transfer_control]].

Dying to hook this up to the assistant, but a cloudy day is forcing me to
curtail further computer use.

----

Also, last night I developed a patch for the hS3 library, that should let
git-annex upload large files to S3 without buffering their whole content in
memory. I have a `s3-memory-leak` in git-annex that uses the new API I
developed. Hopefully hS3's maintainer will release a new version with that
soon.


code:
## today

Came up with four groups of repositories that it makes sense to
define standard preferred content expressions for.

[[!format haskell """
	preferredContent :: StandardGroup -> String
	preferredContent ClientGroup = "exclude=*/archive/*"
	preferredContent TransferGroup = "not inallgroup=client and " ++ preferredContent ClientGroup
	preferredContent ArchiveGroup = "not copies=archive:1"
	preferredContent BackupGroup = "" -- all content is preferred
"""]]

[[preferred_content]] has the details about these groups, but
as I was writing those three preferred content expressions,
I realized they are some of the highest level programming I've ever done,
in a way.

Anyway, these make for a very simple repository configuration UI:

[[!img /assistant/repogroups.png alt="form with simple select box"]]

## yesterday (forgot to post this)

Got the assistant honoring preferred content settings. Although so far that
only determines what it transfers. Additional work will be needed to make
content be dropped when it stops being preferred.

----

Added a "configure" link next to each repository on the repository config
page. This will go to a form to allow setting things like repository
descriptions, groups, and preferred content settings.

----

Cut a release.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlVwbp82f-7rNOyEpvqc9A8FEfn7wl2Akw"
 nickname="Jos"
 subject="Works for me"
 date="2012-10-11T13:40:08Z"
 content="""
These categories make it much clearer to me how I will be able to use git annex assistant
"""]]


code:
Bugfixes all day.

The most amusing bug, which I just stumbled over randomly on my own,
after someone on IRC yesterday was possibly encountering the same issue,
made `git annex webapp` go into an infinite memory-consuming loop on
startup if the repository it had been using was no longer a valid git
repository.

Then there was the place where HOME got unset, with also sometimes amusing
results.

Also fixed several build problems, including a threaded runtime hang
in the test suite. Hopefully the next release will build on all Debian
architectures again.

I'll be cutting that release tomorrow. I also updated the linux
prebuilt tarballs today.

----

Hmm, not entirely bugfixes after all. Had time (and power) to work
on the repository configuration form too, and added a check box to it that
can be unchecked to disable syncing with a repository.
Also, made that form be displayed after the webapp creates a new
repository.


code:
Switched the OSX standalone app to use `DYLD_ROOT_PATH`.
This is the third `DYLD_*` variable I've tried; neither
of the other two worked in all situations. This one may do better.
If not, I may be stuck modifying the library names in each executable
using `install_name_tool`
([good reference for doing that](http://www.mikeash.com/pyblog/friday-qa-2009-11-06-linking-and-install-names.html)).
As far as I know, every existing dynamic library lookup system is broken
in some way other other; nothing I've seen about OSX's so far
disproves that rule.

Fixed a nasty utf-8 encoding crash that could occur when merging the
git-annex branch. I hope I'm almost done with those.

Made git-annex auto-detect when a git remote is on a sever like github
that doesn't support git-annex, and automatically set annex-ignore.

Finished the UI for pausing syncing of a remote. Making the syncing
actually stop still has some glitches to resolve.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmubB1Sj2rwFoVdZYvGV0ACaQUJQyiJXJI"
 nickname="Paul"
 subject="Thank you for stickers"
 date="2012-10-14T13:59:55Z"
 content="""
Mail received in Finland.
"""]]


code:
Did a fair amount of testing and bug fixing today.

There is still some buggy behavior around pausing syncing to a remote,
where transfers still happen to it, but I fixed the worst bug there.

Noticed that if a non-bare repo is set up on a removable drive,
its file tree will not normally be updated as syncs come in -- because the
assistant is not running on that repo, and so incoming syncs are not
merged into the local master branch. For now I made it always use bare
repos on removable drives, but I may want to revisit this.

The repository edit form now has a field for the name of the repo,
so the ugly names that the assistant comes up with for ssh remotes
can be edited as you like. `git remote rename` is a very nice thing.

Changed the preferred content expression for transfer repos to this:
"not (inallgroup=client **and copies=client:2)**". This way, when there's
just one client, files on it will be synced to transfer repos, even
though those repos have no other clients to transfer them to. Presumably,
if a transfer repo is set up, more clients are coming soon, so this avoids
a wait. Particularly useful with removable drives, as the drive will start
being filled as soon as it's added, and can then be brought to a client
elsewhere. The "2" does mean that, once another client is found,
the data on the transfer repo will be dropped, and so if it's brought
to yet another new client, it won't have data for it right away.
I can't see way to generalize this workaround to more than 2 clients;
the transfer repo has to start dropping apparently unwanted content at
some point. Still, this will avoid a potentially very confusing behavior
when getting started.

----

I need to get that dropping of non-preferred content to happen still.
Yesterday, I did some analysis of all the events that can cause previously
preferred content to no longer be preferred, so I know all the places
I have to deal with this.

The one that's giving me some trouble is checking in the transfer scan. If it
checks for content to drop at the same time as content to transfer, it could
end up doing a lot of transfers before dropping anything. It'd be nicer to
first drop as much as it can, before getting more data, so that transfer
remotes stay as small as possible. But the scan is expensive, and it'd also
be nice not to need two passes.


code:
I was mostly working on other things today, but I did do some bug fixing.
The worst of these is a bug introduced in 3.20121009 that breaks
`git-annex-shell configlist`. That's pretty bad for using git-annex
on servers, although you mostly won't notice unless you're just getting
started using a ssh remote, since that's when it calls configlist.
I will be releasing a new version as soon as I have bandwidth (tomorrow).

Also made the standalone Linux and OSX binaries build with ssh connection
caching disabled, since they don't bundle their own ssh and need to work
with whatever ssh is installed.


code:
More bugfixes today. The assistant now seems to have enough users that
they're turning up interesting bugs, which is good. But does keep me too
busy to add many more bugs\^Wcode.

The fun one today made it bloat to eat all memory when logging out of a
Linux desktop. I tracked that back to a bug in the Haskell DBUS library
when a session connection is open and the session goes away. Developed a
test case, and even profiled it, and sent it all of to the library's
author. Hopefully there will be a quick fix, in the meantime today's
release has DBUS turned off. Which is ok, it just makes it a little bit
slower to notice some events.


code:
I released git-annex an unprecidented two times yesterday, because just
after the first release, I learned of a another zombie problem. Turns out
this zombie had existed for a while, but it was masked by zombie reaping
code that I removed recently, after fixing most of the other zombie
problems. This one, though, is not directly caused by git-annex. When rsync
runs ssh, it seems to run two copies, and one is left unwaited on as a
zombie. Oddly, this only happens when rsync's stdout is piped into
git-annex, for progress bar handling. I have not source-dived rsync's code
to get to the bottom of this, but I put in a workaround.

I did get to the bottom of yesterday's runaway dbus library. Got lucky and
found the cause of the memory leak in that library on the first try, which
is nice since each try involved logging out of X. I've been corresponding
with its author, and a fix will be available soon, and then git-annex will
need some changes to handle dbus reconnection.

-----

For the first time, I'm starting to use the assistant on my own personal
git-annex repo. The preferred content and group settings let me configure it
use the complex system of partial syncing I need. For example, I have this
configured for my sound files, keeping new podcasts on a server until they land
somewhere near me. And keeping any sound files that I've manually put on my
laptop, and syncing new podcasts, but not other stuff.

	# (for my server)
	preferred-content 87e06c7a-7388-11e0-ba07-03cdf300bd87 = include=podcasts/* and (not copies=nearjoey:1)
	# (for my laptop)
	preferred-content 0c443de8-e644-11df-acbf-f7cd7ca6210d = exclude=*/out/* and (in=here or (include=podcasts/*))

Found and fixed a bug in the preferred content matching code, where
if the assistant was run in a subdirectory of the repo, it failed to
match files correctly.


code:
[[!comment format=mdwn
 username="http://a-or-b.myopenid.com/"
 ip="203.45.2.230"
 subject="comment 1"
 date="2012-10-19T01:21:42Z"
 content="""
Can you elaborate on the \"nearjoey\" option below?
> (not copies=nearjoey:1)

Are you able to assign a \"nearjoey\" flag to multiple annexes?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.23"
 subject="comment 2"
 date="2012-10-19T17:07:08Z"
 content="""
Yes, using the group command.
"""]]


code:
Got unwanted content to be dropped from the local repo, as well as remotes
when doing the expensive scan. I kept the scan a single pass for now,
need to revisit that later to drop content before transferring more.
Also, when content is downloaded or uploaded, this can result in it
needing to be dropped from somewhere, and the assistant handles that too.

There are some edge cases with hypothetical, very weird preferred
content expressions, where the assistant won't drop content right away.
(But will later in the expensive scan.) Other than those, I think I have
nearly all content dropping sorted out. The only common case I know of where
unwanted content is not dropped by the assistant right away is when a file
is renamed (eg, put in a "Trash" directory).

In other words, repositories put into the transfer group will now work as
described, only retaining content as long as is needed to distribute it to
clients. Big milestone!


code:
A rather frustrating and long day coding went like this:

## 1-3 pm

Wrote a single function, of which all any Haskell programmer needs to know
is its type signature:

	Lsof.queryDir :: FilePath -> IO [(FilePath, LsofOpenMode, ProcessInfo)]

When I'm spending another hour or two taking a unix utility like lsof and
parsing its output, which in this case is in a rather complicated
machine-parsable output format, I often wish unix streams were strongly
typed, which would avoid this bother.

## 3-9 pm

Six hours spent making it defer annexing files until the commit thread
wakes up and is about to make a commit. Why did it take so horribly long?
Well, there were a number of complications, and some really bad bugs
involving races that were hard to reproduce reliably enough to deal with.

In other words, I was lost in the weeds for a lot of those hours...

At one point, something glorious happened, and it was always making exactly
one commit for batch mode modifications of a lot of files (like untarring
them). Unfortunately, I had to lose that gloriousness due to another
potential race, which, while unlikely, would have made the program deadlock
if it happened.

So, it's back to making 2 or 3 commits per batch mode change. I also have a
buglet that causes sometimes a second empty commit after a file is added.
I know why (the inotify event for the symlink gets in late,
after the commit); will try to improve commit frequency later.

## 9-11 pm

Put the capstone on the day's work, by calling lsof on a directory full
of hardlinks to the files that are about to be annexed, to check if any
are still open for write.

This works great! Starting up `git annex watch` when processes have files
open is no longer a problem, and even if you're evil enough to try having
multiple processes open the same file, it will complain and not annex it
until all the writers close it.

(Well, someone really evil could turn the write bit back on after git annex
clears it, and open the file again, but then really evil people can do
that to files in `.git/annex/objects` too, and they'll get their just
deserts when `git annex fsck` runs. So, that's ok..)

----

Anyway, will beat on it more tomorrow, and if all is well, this will finally
go out to the beta testers.


code:
[[!comment format=mdwn
 username="http://dieter-be.myopenid.com/"
 nickname="dieter"
 subject="comment 1"
 date="2012-06-16T09:14:26Z"
 content="""
maybe at some point, your tool could show \"warning, the following files are still open and are hence not being annexed\"
to avoid any nasty surprises of a file not being annexed and the user not realizing it.
"""]]


code:
Got preferred content checked when files are moved around.
So, in repositories in the default client group, if you make a
"archive" directory and move files to it, the assistant will drop
their content (when possible, ie when it's reached an archive or backup).
Move a file out of an archive directory, and the assistant will get its
content again. Magic.

Found an intractable bug, obvious in retrospect, with the git-annex branch
read cache, and had to remove that cache. I have not fully determined
if this will slow down git-annex in some use cases; might need to add more
higher-level caching. It was a very minimal cache anyway, just of one file.

Removed support for "in=" from preferred content expressions. That was
problimatic in two ways. First, it referred to a remote by name, but
preferred content expressions can be evaluated elsewhere, where that remote
doesn't exist, or a different remote has the same name. This name lookup
code could error out at runtime. Secondly, "in=" seemed pretty useless, and
indeed counterintuitive in preferred content expressions. "in=here" did not
cause content to be gotten, but it did let present content be dropped.
Other uses of "in=" are better handled by using groups.

In place of "in=here", preferred content expressions can now use "present",
which is useful if you want to disable automatic getting or dropping of
content in some part of a repository. Had to document that "not present"
is not a good thing to use -- it's unstable. Still, I find "present" handy
enough to put up with that wart.

Realized last night that the code I added to the TransferWatcher
to check preferred content once a transfer is done is subject to a race;
it will often run before the location log gets updated. Haven't found a good
solution yet, but this is something I want working now, so I did put in a
quick delay hack to avoid the race. Delays to avoid races are never a real
solution, but sometimes you have to TODO it for later.

----

Been thinking about how to make the assistant notice changes to configuration
in the git-annex branch that are merged in from elsewhere while it's running.
I'd like to avoid re-reading unchanged configuration files after each merge
of the branch.

The most efficient way would be to reorganise the git-annex branch, moving
config files into a configs directory, and logs into a logs directory. Then it
could `git ls-tree git-annex configs` and check if the sha of the configs
directory had changed, with git doing minimal work
(benchmarked at 0.011 seconds).

Less efficiently, keep the current git-annex branch layout, and
use: `git ls-tree git-annex uuid.log remote.log preferred-content.log group.log trust.log`
(benchmarked at 0.015 seconds)

Leaning toward the less efficient option, with a rate limiter so it
doesn't try more often than once every minute. Seems reasonable for it to
take a minute for config changes take effect on remote repos, even
if the assistant syncs file changes to them more quickly.


code:
Added yet another thread, the ConfigMonitor. Since that thread needs to run
code to reload cached config values from the git-annex branch when files
there change, writing it also let me review where config files are cached,
and I found that every single config file in the git-annex branch does
get cached, with the exception of the uuid.log. So, added a cache for that,
and now I'm more sanguine about yesterday's removal of the lower-level
cache, because the only thing not being cached is location log information.

The ConfigMonitor thread seems to work, though I have not tested it
extensively. The assistant should notice and apply config changes
made locally, as well as any config changes pushed in from remotes.
So, for example, if you add a S3 repo in the webapp, and are paired with
another computer, that one's webapp will shortly include the new repo in
its list. And all the preferred content, groups, etc settings will
propigate over and be used as well.

Well ... almost. Seems nothing causes git-annex branch changes to be
pushed, until there's some file change to sync out.


code:
Time to solve the assistant's [[cloud]] notification problem. This is
really the last big road-bump to making it be able to sync computers
across the big, bad internet.

So, IRC still seems a possibility, but I'm going to try XMPP first. Since
Google Talk uses XMPP, it should be easy for users to get an account, and
it's also easy to run your own XMPP server.

Played around with the Haskell XMPP library. Clint helpfully showed me an
example of a simple client, which helped cut through that library's thicket
of data types. In short order I had some clients that were able to see each
other connecting to my personal XMPP server. On to some design..

1. I want to avoid user-visible messages.
   (dvcs-autosync also uses XMPP, but I checked the code and it
   seems to send user-visible messages, so I diverge from its lead here.)
   This seems very possible, only a matter of finding the right
   way to use the protocol, or an appropriate and widely deployed extension.
   The only message I need to send with XMPP, really, is "I have pushed to our
   git repo". One bit of data would do; being able to send a UUID of the repo
   that did the update would be nice.

2. I'd also like to broadcast my notification to a user's buddies.
   dvcs-autosync sends only self-messages, but that requires every node
   have the same XMPP account configured. While I want to be able to run
   in that mode, I also want to support pairs of users who have their own XMPP
   accounts, that are buddied up in XMPP.

3. To add to the fun, the assistant's use of XMPP should not make that XMPP
   account appear active to its buddies. Users should not need a dedicated
   XMPP account for git-annex, and always seeming to be available when
   git-annex is running would not be nice.

The first method I'm trying out is to encode the notification
data inside a XMPP presence broadcast. This should meet all three
requirements. The plan is two send two
presence messages, the first marks the client as available, and the second
as unavailable again.
The "id" attribute will be set to some
value generated by the assistant. That attribute is allowed on presence
messages, and servers are [required to preserve it](http://xmpp.org/rfcs/rfc6121.html#presence-probe-inbound-id)
while the client is connected.
(I'd only send unavailable messages, but while
that worked when I tested it using the prosody server, with google talk,
repeated unavailable messages were suppressed. Also, google talk does not
preserve the "id" attribute of unavailable presence messages.)

If this presence hackery doesn't work out, I could try
[XEP-0163: Personal Eventing Protocol](http://xmpp.org/extensions/xep-0163.html).
But I like not relying on any extensions.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmQ4Oe5-qOANRuZel9kDvtBfQG1zlEcIzw"
 nickname="Dominik"
 subject="PEP is ok"
 date="2012-10-24T05:15:38Z"
 content="""
PEP and pubsub are really widely used. You will have a hard time finding a server that doesn't support it.

Also note XMPPs ability to send status to only selected buddies and even resources. You can globally appear as offline while sending availability to myownjid@example.com/thatannexthere.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmQ4Oe5-qOANRuZel9kDvtBfQG1zlEcIzw"
 nickname="Dominik"
 subject="Remote control clients"
 date="2012-10-24T05:18:02Z"
 content="""
Then of course there is XEP-0146, which has some advantages to it, like controlling annex from the context menu of another Jabber client.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.23"
 subject="comment 3"
 date="2012-10-24T22:44:41Z"
 content="""
Google talk does not seem to support PEP. At least, I'm sending something that prosody accepts and that looks like the example at <http://xmpp.org/extensions/xep-0163.html#howitworks>, and it replies with an IQ Error.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmaM3-vbAh5B8tY_IL3yh31ik6cAVsgoPw"
 nickname="IC"
 subject="Haskell XMPP"
 date="2012-10-27T05:54:53Z"
 content="""
Which xmpp library you've decided to use?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4978:f:21a::2"
 subject="comment 5"
 date="2012-10-27T06:08:44Z"
 content="""
I'm using http://hackage.haskell.org/package/network-protocol-xmpp

While its author thinks \"its current design (session in an XMPP monad) is a failed mistake\", I've found it pretty good. Aside from the segfault problem, which we got fixed today.
"""]]


code:
Built out the XMPP push notifier; around 200 lines of code.
Haven't tested it yet, but it just might work. It's in the `xmpp` branch
for now.

I decided to send the UUID of the repo that was pushed to, otherwise
peers would have to speculatively pull from every repo. A wrinkle in this
is that not all git repos have a git-annex UUID. So it might notify that
a push was sent to an unidentified repo, and then peers need to pull from
every such repo. In the common case, there will only be one or a few such
repos, at someplace like at github that doesn't support git-annex. I could
send the URL, but there's no guarantee different clients have the same
URLs for a git remote, and also sending the URL leaks rather more data than
does a random UUID.

Had a bit of a scare where it looked like I couldn't use the haskell
`network-protocol-xmpp` package together with the `mtl` package that
git-annex already depends on. With help from #haskell I found the way
to get them co-existing, by using the PackageImports extension. Whew!

Need to add configuration of the XMPP server to use in the webapp, and
perhaps also a way to create `.git/annex/creds/notify-xmpp` from the
command line.