code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn4yApngiiAJE5iX1MQ0VO_LXjLt4LgdHE"
 nickname="Jim"
 subject="No mass storage"
 date="2012-12-02T03:33:06Z"
 content="""
I would have asked for better FAT support, but my Galaxy Nexus doesn't support mass storage mode, and MTP is not as easy to use nor reliably supported everywhere.  So I'd lean towards functionality that stays on the phone as much as possible.
"""]]


code:
What directory should the Android webapp default to creating an annex in?

Same as the desktop webapp, users will be able to enter a directory they
want the first time they run it, but to save typing on android, anything
that gets enough votes will be included in a list of choices as well.

[[!poll open=yes expandable=yes 68 "/sdcard/annex" 6 "Whole /sdcard" 7 "DCIM directory (photos and videos only)" 2 "Same as for regular git-annex. ~/annex/"]]

repo_name: kakafarm/git

license: unknown


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 1"
 date="2013-04-19T06:29:14Z"
 content="""
While my main use case is a photo repo, I don't think it's wise to default to there.

Similarly, spamming what amounts to the root dir of \"external\" storage with repos seems like a bad idea.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnE6kFAbud1LWrQuyX76yMYnUjHt9tR-A8"
 nickname="Leonardo"
 subject="Direct mode"
 date="2013-04-20T20:55:09Z"
 content="""
I think direct mode should be the default on Android
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 3"
 date="2013-04-20T23:38:47Z"
 content="""
@Richard, including all of /sdcard seems a reasonable thing to do if you want to back it all up. I don't know how likely anyone would be to want to sync the whole contents though.

@Leonardo direct mode is the default for all respositories created by the webapp, as well as all repositories on crippled filesystems that cannot support indirect mode. Android webapp thus defaults to direct mode twice over!
"""]]


code:
What should I work on in April? I expect I could get perhaps two of these
features done in a month if I'm lucky. I have only 3 more funded months,
and parts of one will be spent working on porting to Windows, so choose wisely!
--[[Joey]]

[[!poll open=yes expandable=yes 4 "upload and download rate limiting" 15 "get webapp working on Android" 5 "deltas: speed up syncing modified versions of existing files" 8 "encrypted git remotes using git-remote-gcrypt" 0 "add support for more cloud storage remotes" 19 "don't work on features, work on making it easier to install and use" 2 "Handle duplicate files" 6 "direct mode (aka real files instead of symlinks) [already done --joey]" 3 "start windows port now"]]

References:

* [[rate_limiting]]
* [[Android]]
* [[deltas]] to speed up syncing modified files (at least for remotes using rsync)
* [[encrypted_git_remotes]]
* [[more_cloud_providers]] (OpenStack Swift, Owncloud, Google drive,
  Dropbox, Mediafire, nimbus.io, Mega, etc.)
* [[old poll on "what is preventing me from using git-annex assistant"|what_is_preventing_me_from_using_git-annex_assistant]]
  (many of the items on it should be fixed now, but I have plenty of bug reports to chew on still)


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm5iosFbL2By7UFeViqkc6v-hoAtqILeDA"
 nickname="Laszlo"
 subject="Duplicate files"
 date="2013-04-01T07:58:51Z"
 content="""
Having duplicate files is fairly easy.
a) Just a backup of your files, to not have accidental deleting.

b) Some programs implements UNDO command (Ctrl-Z) as simply copying over the working text file
to a backup directory.

c) If you tests precompiled programs, like git annex itself, it has identical files across releases.

For more info about duplicate files, read hear these two bugreports:

http://git-annex.branchable.com/bugs/When_syncing_two_repositories__44___git_annex_uses_9x_times_diskspace/
(last comment)

http://git-annex.branchable.com/bugs/Direct_mode_keeps_re-checksuming_duplicated_files/
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkWG4T4SGZxY-q4Wo8Jbxwb67g4J-WYmQM"
 nickname="Mark"
 subject="don't work on features option"
 date="2013-04-01T14:03:35Z"
 content="""
The \"don't work on features...\" poll entry is a bit vague, so I figured stating my interpretation of it and why I choose it might be a good idea.
I've had git-annex installed for the last few months during which it has been steadily improving, but IMO it still lacks polish.
From time to time I see transient issues: (even on the current version)

- random download stalling when I know the remote is accessible
- files disappearing then reappearing without an obvious cause
- xmmp hanging

I do not have any means of replicating these issues (though IIRC some of the recently worked on bugs related to these issues).
In my past experience this indicates that there are all sorts of 'fun' bugs hiding in the source which you seem to be chasing down.
Heck, I could have a simple configuration error from when I set things up on my remote server.
There was little documentation available when I setup my remote server for this and unless I have missed something in the RecentChanges feed, there still is relatively little.
So, some issues with my xmmp daemon, local ssh keys via ssh-agent, or bad $PATH stuff could be causing things to subtlety malfunction at no fault of git-annex.

Falling back to the command line only tends to be a good response, but outside of the assistant there does not seem to be any manual way to handle the special remotes.
Fair enough, but it would seem logical for error handling to recognize that these are assistant only urls rather than some generic \"bad url\".
Just getting the right error messages while a small touch would be a sign of some polish even though it is a nitpick.

Perhaps some simple test code run repeatedly to form a stress test could reveal some odd behavior, but I'm not sure myself.

With all of that said, I like what has been done so far and I'm hoping to see all the nooks of git-annex get the polishing that they deserve.
"""]]


code:
[[!comment format=mdwn
 username="http://adamspiers.myopenid.com/"
 nickname="Adam"
 subject="Ask for more money :-)"
 date="2013-04-01T19:01:49Z"
 content="""
This is slightly off topic, but from my perspective this Kickstarter project has been a riotous success, and due to the credibility you have built up from 9 months of consistently high quality work produced in a very transparent and community-oriented fashion, I'd be surprised if you weren't able to raise more money to continue development after the 12 months is up.  Assuming you want to of course :)  Maybe you need a break or change of scene.  But if not, it would be a shame not to continue if the userbase wants it.
"""]]


code:
Background: git-annex supports storing data in various [[special remotes]].
The git-annex assistant will make it easy to configure these, and easy
configurators have already been built for a few: removable drives, rsync.net,
locally paired systems, and remote servers with rsync.

Help me prioritize my work: What special remote would you most like
to use with the git-annex assistant?

[[!poll open=yes 16 "Amazon S3 (done)" 12 "Amazon Glacier (done)" 10 "Box.com (done)" 73 "My phone (or MP3 player)" 25 "Tahoe-LAFS" 13 "OpenStack SWIFT" 34 "Google Drive"]]

This poll is ordered with the options I consider easiest to build
listed first. Mostly because git-annex already supports them and they
only need an easy configurator. The ones at the bottom are likely to need
significant work. See [[cloud]] for detailed discussion.

Have another idea? Absolutely need two or more? Post comments..


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 1"
 date="2012-09-13T08:39:59Z"
 content="""
I've been looking at ceph for various reasons in work, it supports a swift interface as well as it's own restful api. so +1 for swift (and any s3 compatible api).
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 2"
 date="2012-09-13T09:07:02Z"
 content="""
Swift has its own API but offers a S3 compatibility layer. Last I tried that layer, it did not work.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmsz4weoPXV2oEtv3zpo9dOxn_SEPz-7Iw"
 nickname="Zooko"
 subject="reasons to like Tahoe-LAFS as a special remote"
 date="2012-10-12T18:17:42Z"
 content="""
Here are a couple of things which are (I think) unique about the Tahoe-LAFS special remote:

1. encryption ; All of the data is encrypted before leaving your local system and heading for the server (or for the clouds). This is true even though you don't (I think) have to enter an encryption key into git-annex to access your data.

(Note: the above implies that you're in danger of permanently losing access to your data, by losing the last copy of the encryption key, if your local git-annex state is lost. This deserves careful consideration.)

2. erasure-coding ; You can configure Tahoe-LAFS to spread the data out in a RAID-like way across multiple remote storage servers, where each server holds only, say, 1/3 of the data, but there are, say, 10 different servers, where any 3 of them are sufficient to give you full access to your data. Does that make sense it uses less bandwidth and storage space than replication (i.e. putting a complete replica of your data on each of 4 or 5 or 10 different storage servers), but it is more robust than sharding (i.e. putting 1/3 of your data on each of three different servers so that if any one of them goes down you lose 1/3 of your data).
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnjrMQEhzd8xI81V9BL2jsKlNgVJLD7PKs"
 nickname="Pankaj"
 subject="Google Music as remote?"
 date="2012-10-31T06:07:32Z"
 content="""
A dropbox like folder which syncs with Google Music. Google Music allows uploading upto 20K songs. Also using git-annex , I can ensure that I dont need to store all the duplicate mp3 on my local drive. Only one copy of \"older music\" is \"arvhived\" on GMusic, whereas more recent songs are on my local drive.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm5iosFbL2By7UFeViqkc6v-hoAtqILeDA"
 nickname="Laszlo"
 subject="comment 10"
 date="2013-01-11T21:57:29Z"
 content="""
@joey: My usecase is fairly simple.
I have a working directory where I do work
(not programing, but text editing with homecooked spreadsheet/texteditor).

I work/type on spot/on site. So I'm walking while typing.

Its easy to do stupid things while you work,
so would be really handy to have a backup automagically on the same computer to prevent mistakes, user errors.
(The last mistake, I did is I copied all my daily files into a \"tmp\" directory,
then I launched my homebrew program which also created a tmp dir then deleted the whole content of it.
I fixed it, but my datas was lost. It is a simple user mistake which could be prevented if I would have a
live backup all the time. Creating a git repo for daily typing is just too much overhead.
It is like you create a git repo for your text messages...)

So my usecase is a live backup. Or the broken file deleting thing on unix/linux since forever.

Laszlo

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 11"
 date="2013-01-11T22:17:07Z"
 content="""
@dzsino: You may want to have a look at [metamonger](https://github.com/RichiH/metamonger) which tries to solve exactly this problem. It's not finished yet, though.

-- Richard
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkx5V3MTbzCXS3J7Mn9FEq8M9bPPYMkAHY"
 nickname="Pedro"
 subject="It fails silently and leaves symlinks behind"
 date="2013-01-11T23:49:51Z"
 content="""
I dedicated a day to looking into using the assistant and although it's clearly an exciting piece of software it's still pretty hard to get working. To get two machines to sync with one another the options seem to be:

1. Connect both of them to a XMPP server: That makes metadata sync flawlessly and yet leaves a symlink mess behind. As it turns out that's because XMPP syncs only metadata but nowhere was that stated on the webapp (that I could find) and no error message given
2. So then I tried the sync method with SSH.

For that you'll need to:

* Have SSH servers installed on both endpoints (for me that was easy to do but I'm sure most users will have a harder time)
* Make sure both computers are actually on the same broadcast network otherwise they won't see each other (I was testing with a VM behind virtualbox's NAT so initially it didn't work)
* Install the assistant on both sides and make sure both the base program and the shell wrapper are in the PATH. (this was harder to do than needed as in the standalone bundle they're not on the same folder and the shell wrapper needs to be run by the runshell script, so even if you get it in the path it will still fail with missing libraries. An Ubuntu PPA would have solved this I'm sure.

For some reason I couldn't figure out they still couldn't sync and symlink mess ensued.

I think there are a few usability issues here:

1. Direct mode should never overwrite files in a way you can no longer use them just because metadata updates came before content updates. I'd rather have an older version of the file than lose access to it.
2. Error reporting needs to be a lot more explicit to be able to debug these issues
3. At the best of times SSH sync is going to be a problem. In my opinion to be really dropbox-like the sync method for normal users should be to connect all devices to the same XMPP account (this works great now) and then have those assistants automatically reach each other by a direct connection when possible and by something like STUN if NAT punching is needed. The SSH/bup/etc remotes all have their place for advanced setups but being able to tell the user to just connect to XMPP and be done would turn this from \"something I'll spend an afternoon writing puppet manifests to deploy in my personal servers/computers\" to \"something I can explain to a non-technical friend how to use over a phone call and then share files with him\".

On the XMPP side though, it would work best if you could authenticate a Google XMPP acount through an OAuth workflow instead of asking for the password. I wouldn't give any program my gmail password but I'd easily give it permission to use gtalk on my behalf.

As I started by saying the assistant (and git-annex in general) is an extremely impressive piece of software that I'm very excited about. I hope it keeps improving at the current pace as there are still a few features I'd love to have (partial content particularly) but by far the thing that's keeping me most from using it is the ability to easily get a reliable sync going that won't leave me with a folder full of symlinks that I don't know what to do about. One of the great things about dropbox is that whenever it isn't running or can't sync for any reason your folder degenerates into a folder like any other on your disk. The right way of thinking about direct mode should be that one, \"a folder like any other that we stream changes from and to on multiple machines\". Right now, with the symlink replacement when sync isn't possible and to a lesser extent the adding of .git folders it doesn't live up to that guarantee.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm5iosFbL2By7UFeViqkc6v-hoAtqILeDA"
 nickname="Laszlo"
 subject="comment 13"
 date="2013-01-12T08:16:48Z"
 content="""
@Pedro: That is an excellent test, and usecase!

Indeed, the gui needs a lot more info on it.
Also file integratity is a real fear with this kind of application.

"""]]


code:
[[!comment format=mdwn
 username="https://launchpad.net/~rubiojr"
 nickname="rubiojr"
 subject="git-annex PPA"
 date="2013-01-15T14:16:35Z"
 content="""
Hey Joey,

I do maintain a PPA with a more up2date version in fact:

https://launchpad.net/~rubiojr/+archive/git-annex

Now that the direct mode is in place, I guess it's probably a good time to backport again.

Thanks for such a great piece of software.
"""]]


code:
[[!comment format=mdwn
 username="https://launchpad.net/~rubiojr"
 nickname="rubiojr"
 subject="direct mode still 'untrusted'?"
 date="2013-01-15T14:23:22Z"
 content="""
BTW,

Does this still apply?

\"I'd like git annex direct to set the repository to untrusted, but I didn't do it. Partly because having git annex indirect set it back to semitrusted seems possibly wrong -- the user might not trust a repo even in indirect mode. Or might fully trust it. The docs will encourage users to set direct mode repos to untrusted -- in direct mode you're operating without large swathes of git-annex's carefully constructed safety net. (When the assistant later uses direct mode, it'll untrust the repository automatically.)\"

http://git-annex.branchable.com/design/assistant/blog/day_151__direct_mode_toggle/
"""]]


code:
[[!comment format=mdwn
 username="http://adamspiers.myopenid.com/"
 nickname="Adam"
 subject="just one thing"
 date="2013-01-16T01:45:54Z"
 content="""
http://git-annex.branchable.com/todo/wishlist:_disable_automatic_commits/ is the only thing I'm waiting for.  Will become a very keen beta-tester (maybe even coder) when that happens.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.7.238"
 subject="comment 17"
 date="2013-01-16T19:38:36Z"
 content="""
@rubiojr I recommend configuring direct mode repos as untrusted if you care about accessing old versions of your files as stored in git. If that is not a concern, direct mode is safe, it's just that it allows editing/deleting any file at any time, even if that's the only copy of the file.
"""]]


code:
[[!comment format=mdwn
 username="http://christian.amsuess.com/chrysn"
 nickname="chrysn"
 subject="comment 18"
 date="2013-01-17T12:04:34Z"
 content="""
the \"I haven't always well understood the differences between commandline operation & the assistant, so the differences would confuse me, and I found the command line more understandable & less scary. Now trying to learn to like & trust the assistant.\" fits pretty well for me -- i'd like to use the webapp at least for viewing, but i'm worried that some auto-magic would kick in, and i regularly have uncommitted stuff in at least one git-annex repo.

a --dry-run flag or similar on the `git annex webapp` would be nice, or a description of what would happen or how to turn that off / make it create other branches / even commit or push something.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnmvVKc1ECzZx7EhtLHBP6RWPZewq4x_9I"
 nickname="Daniele"
 subject="Windows + git ignore like syntax for files to sync"
 date="2013-04-19T10:47:36Z"
 content="""
Hi,

my company would need this for some project.

But to adopt it I would need:
* windows support (I don't care if now symlink support for now)
* being able to define which files are automatically synced using a syntax similar to the gitignore one
* being able to disable auto-commit (just synching, user should be able to commit on their own)

thanks

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="Two reasons"
 date="2013-01-10T18:00:05Z"
 content="""
As of right now, the assistant has only secondary benefit over plain git-annex inasmuch there's a lot more code activity.

As soon as the assistant supports Android, I will use for syncing photos off of my phone and may start to use the assistant on my usual repos as a natural consequence.

Additionally, there's a subjective feeling of more bugs being reported. That may or may not be true, but as long as there's no Android port, I don't have an actual reason to \"risk\" it.

-- Richard

PS: This seems to be the first poll where you can place only one vote while it's the first where I really wanted to vote on two separate items.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 20"
 date="2013-04-19T18:01:44Z"
 content="""
You can turn off automatic commits in the webapp by pausing syncing for the local repository.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmH-n1yD04qmSeXKKzYaXFTN1ciFWVb5As"
 nickname="Martyn"
 subject="Can't seem to get the configuration I am looking for from the webapp"
 date="2014-01-02T23:41:44Z"
 content="""
Currently, I have a local server, desktop and laptop (all on the same network). My laptop leaves the house occasionally where I work on it...
Quite simply, I want make the local server the centralised repository and have the desktop + laptop sync up and down to and from the server. The server would act as a backup and file change history between all local repositories on desktop and laptop. So I would sync between trips away.

Sadly every combination with git-annex webapp fails for a number of reasons, it could be just me but here is what I've experienced:

a) I want to use direct mode ALL the time or at least be able to configure it without the command line. Currently server side set ups are indirect depending on the repository group. So no symlinks basically. I think \"full backup\" and \"client\" are the only two that do what I was hoping for here.
b) The list of repository group types miss something I'm looking for, I want the \"full backup\" with \"transfer\" for the remote server - I guess?
c) For remote cloud solutions, I would like the files to exist, not just the raw .git/ directory to exist on the server. When I tested it, I couldn't see a way to test on the server itself the integrity of the files (could be my lack of git-annex knowledge :)
d) I would like it if the assistant made it simpler to configure remotes, i.e. right now if I have 2 directories I believe I would need 4 configurations to push/pull locally and remotely to the 1 server with each machine. I've had this working and it's the closest I've got so far, but tedious if I want to do it for 5 directories on 2 machines (that's 20 configurations to add in all unless there is an easier way?)
e) The \"share with other devices\" option seems one of the best right now, but it suffers with the issues of c) AFAICT. Why not have the server just be a remote when unencrypted?

From some reading up on the command line tools, it seems like it would be easier to set up using those for what I want than using the webapp.
I currently love the webapp, it's just missing some polish and options to do what I need.

Thanks for the great work so far! :)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawniCRkhl_W87gOK5eElfsef3FoUsUFpAr4"
 nickname="Alexandre"
 subject="Two assistants with shared transfer repository"
 date="2013-01-11T07:17:12Z"
 content="""
I voted for the Windows port, which unfortunately I must use at work.

But another thing that Dropbox gets right is that you can [sync two clients with one shared repository](http://git-annex.branchable.com/forum/How_to_set_up_two_assistants_with_one_shared_transfer_repository__63__) that works without any configuration whether you are home or somewhere else. This use case is covered by sync over XMPP, but there are two small issues with this :
- if I use Google XMPP servers to sync two assistants in the same network, everything goes through the slower Internet link
- if I setup my own XMPP server, well, this is more setup, more open ports, etc.

It would be nice to have repository pushs on a shared repository notified to the connected clients in the SSH connection.
"""]]


code:
[[!comment format=mdwn
 username="http://a-or-b.myopenid.com/"
 ip="220.244.41.108"
 subject="comment 3"
 date="2013-01-11T09:32:33Z"
 content="""
I've got a few wishes for git-annex…  Referring to the poll options:

- I need an Android port

I think this is the biggest issue for getting a high WAF.  (I want to put our photos in an annex, and getting them auto-magically off her phone would get me massive brownie points.)

- Initially the lack of direct-mode. Now concerns about the safety of direct mode. Perhaps after the next release.

The \"potentially unsafe\" sort of comments in the blog make me worry about trusting my data to the direct mode.  Saying that I *really* appreciate the honesty of the comments.  That is why I want to keep using, and recommend to others, git-annex.

- I haven't always well understood the differences between commandline operation & the assistant, so the differences would confuse me, and I found the command line more understandable & less scary. Now trying to learn to like & trust the assistant.

I also have worried about the automatic adding by the assistant - prior to direct mode.

- throttling transfers, it upsets people when I saturate the connection

I think a bittorrent option would be ideal.  Not public torrents, but torrents between my annexes.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm5iosFbL2By7UFeViqkc6v-hoAtqILeDA"
 nickname="Laszlo"
 subject="Two local repositories"
 date="2013-01-11T10:31:54Z"
 content="""
Here is my usecase:
I want two local (direct mode) repositories. So basically a working directory and a backup directory to the working one.

The removable media repository *almost* fulfill this usecase, except you can only select drive and not a directory.
I think it would be relatively simple to implement. (A drive is a directory under linux after all).

Best,
 Laszlo

"""]]


code:
[[!comment format=mdwn
 username="marvin"
 ip="91.154.225.32"
 subject="SSH keys"
 date="2013-01-11T12:02:50Z"
 content="""
Really looking forward to seeing this project go forward and impressed by what you have accomplished.

I have my own server which I want to use as a repository, so I'll choose \"Remote server\". I want to use direct mode.
Now I can only create an encrypted remote which is fine for me, but I don't know how to sync that with another client using this remote.

Also with the generated ssh keys I no longer can log into my server normally as it tries to use the generated keys.
The generated keys should also be restricted to only be able to change the repository and nothing else.

Maybe webdav could be used to create something with better access control and repositories for groups etc.
Also with a simple web upload/download page, similar to dropbox.

Sorry if my post is long and idiotic, just simple (hopefully constructive) feedback from a simple user :)
"""]]


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="173.162.44.162"
 subject="comment 6"
 date="2013-01-11T15:06:47Z"
 content="""
Laszlo's concern about \"you can only select a drive and not a directory\" relates to a problem I had.

I have two local directories, ~/annex indirect my main annex, and ~/directannex for playing with direct mode.

I had hoped to back each up to the same shared drive, but didn't realize there was no option to select a directory within a shared drive, so when I tried to do that through the assistant I inadvertently connected the two by backing them up to the same directory on the shared drive!

This is probably a \"power user problem\" which has a \"power user solution\" of setting up the separate repository manually.  But it did take me by surprise.

"""]]


code:
[[!comment format=mdwn
 username="https://dzsino.myopenid.com/"
 nickname="dzsino"
 subject="Recording file creation dates?"
 date="2013-01-11T16:35:27Z"
 content="""
I've just started using git annex, command line feels safer for the time being. One issue I've got with assistant that I had to kill it couple of times when it got confused over something (could be that I used command line git annex at the same time?),  shutdown from the UI is welcome.

More of a git annex issue for me that I lost file creation dates for annexed files, which i don't really mind for most of my content, except really old photos without proper EXIF tags.
I think of git annex as more of a DVR for my digital life, rather than a mere sync tool, so I would welcome some basic metadata facility. Would you consider adding this?
(Really getting carried away, I would want to full text search on file names, ID3 tags, document metadata and text etc. and requesting files based on search hits, spotlight style..)
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.7.238"
 subject="comment 8"
 date="2013-01-11T20:02:39Z"
 content="""
@Alexandre, what you want can be achieved by setting up XMPP pairing, and also [[local pairing|assistant/local_pairing_walkthrough]]. Then when computers are on the same network, transfers are done via the LAN.

@Laszlo, this can be set up without using the webapp. Just make the two repositories, using the webapp. Then go into each and run \"git remote add myotherrepo $path_to_other_repo\". Then assistant will automatically sync them. I *have* thought about adding a configurator for this to the webapp. It would help to know what use cases you're getting at with this.

@marvin The generated ssh keys are configured to only allow running `git-annex-shell`, which limits it to only acting on the repository. The keys are also configured, in `.ssh/config` to only be used when a particular host alias is used. They should not be used when you just ssh normally to the host. If this is happening to you, please file a bug report with details.

@dzsino you can get file creation dates out of git using git log :)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkQafKy7hNSEolLs6TvbgUnkklTctUY9LI"
 nickname="Zellyn"
 subject="Question about git-annex"
 date="2013-01-11T21:05:55Z"
 content="""
Not sure where the correct place to ask this is...

My use case for git-annex is to back up important files (pictures, videos, etc.) on a hard disk at work and one at home. Is there an easy way to do that? I'd like to be able to get git-annex to fill up my phone/laptop with files that it thinks are backed up in <2 of the locations, and ferry them back and forth. They can be deleted to free up space after that, if necessary.

Thanks for all your work, by the way. I missed the kickstarter, but if there's a simple and direct way to donate (dwolla?, gittip?), I'd love to chip in.
"""]]


code:
Currently, git-annex takes a very lazy approch to displaying
progress into. It just lets rsync or whatever display the progress
for it, in the terminal.

Something better is needed for the [[webapp]]. There needs to be a
way for the web app to know what the current progress is of all transfers.

This is one of those potentially hidden but time consuming problems.

## downloads

* Watch temp file as it's coming in and use its size.
  Can either poll every .5 seconds or so to check file size, or
  could use inotify. **done**
* When easily available, remotes call the MeterUpdate callback as downloads
  progress. **done**
* S3 TODO
  While it has a download progress bar, `getObject` probably buffers the whole
  download in memory before returning. Leaving the progress bar to only
  display progress for writing the file out of memory. Fixing this would
  involve making hS3 stream better (also avoids it wasting memory).

## uploads

Each individual remote type needs to implement its own support for calling
the MeterUpdate callback as the upload progresses.

* git: **done**
* rsync: **done**
* directory: **done**
* web: Not applicable; does not upload
* webdav: **done**
* S3: **done**
* glacier: **done**
* bup: TODO
* hook: Would require the hook interface to somehow do this, which seems
  too complicated. So skipping.

## communication

It may be worth using a better communication channel than files on disk for
the transfer progress. Shared memory could be used, or http posts to the
webapp.


code:
[[!comment format=mdwn
 username="http://abhidg.myopenid.com/"
 ip="129.67.132.87"
 subject="librsync"
 date="2012-06-13T02:14:29Z"
 content="""
There's librsync which might support reporting the progress through its API, but it seems to be in beta.
"""]]


code:
Webapp needs a simple speed control knob, especially to avoid saturating
bandwidth on uploads.

We have basic throttling support in git-annex for rsync,
but none for any special remotes. A good first step would be to expose
this in the webapp, and ensure that `git-annex-shell` also honors it when
sending/receiving data.

We actually need two speed controls, one for upload and one for download.

It is probably not necessary to throttle git push/pull operations, as the
data transfers tend to be small. Only throttling file transfers is
essential.

## possibility: trickle

Since `git-annex transferkeys` is a separate process, one easy option would
be to run it inside `trickle`. If the user changes the bandwidth limits,
it could kill the transfer and restart it with different trickle options.

Problem: Not all special remotes support resuming transfers, so this is
suboptimal. (So too are the pause/resume buttons, when using those
remotes!)

`trickle` is available for OSX as well as Linux and BSDs.
<http://hints.macworld.com/comment.php?mode=view&cid=65362>
<http://mac.softpedia.com/get/Network-Admin/Trickle.shtml>
It is probably not easily available for Android, as it uses `LD_PRELOAD`.

## possibility: built in IO limiting

A cleaner method would be to do the limiting inside git-annex. We already
have metered file IO. It should be possible to make the meter not only report
on the transfer speed, but detect when it's going too fast, and delay. This
will delay the file streaming through the special remote's transfer code,
so should work for a variety of special remotes. (Not for rsync or bup
or git-annex-shell though, so those need to be handled separately.)

Should work well for uploads at least. I don't know how well it would work
for throttling downloads; the sender may just keep sending data and the
data buffer before it gets to the IO meter. Maybe once the buffers fill the
OS would have the TCP throttled down. Needs investigation; trickle claims
to throttle downloads.

## communications channels

There would need to be a communication channel for the assistant to tell
`git annex transferkeys` when the rate limit has changed. It could for
example send it a SIGUSR1, and then leave it up to the process to reload
the git config. Inside the IO meter, we could have an MVar that contains
the current throttle value, so the IO meter could check it each time it's
called and adjust its throttling appropriately.

Ideally, the assistant could also communicate in the same way with
`git-annex-shell` to tell it when the limit has changed. Since
`git-annex-shell` uses rsync, it would need to abort the transfer, and rely
on the other side retrying to start it up with the new limit.


code:
Currently the assistant sets up dedicated ssh keys, that can just use
git-annex. This is ok. The problem is that the initial 2 connections to the
ssh server when setting up these keys involve a password prompt, which is
done at the console unless the system happens to have a working ssh agent
that can pop up a dialog. That can be confusing.

It would be nice to have the webapp prompt for the password. Can it be done
securely?

This might come down to a simple change to the webapp to prompt for the
password, and then rather a lot of pain to make the webapp use HTTPS so we
can be pretty sure noone is sniffing the (localhost) connection.

## ssh-askpass approach

* If ssh-askpass is in PATH, or `SSH_ASKPASS` is set, do nothing.
  (Unless webapp is run remotely.)
  XXX not currently done; the UI would need to omit the password entry
  fields in this case.
* Otherwise, have the assistant set `SSH_ASKPASS` to a command that will
  cause the webapp to read the password and forward it on. Also, set
  DISPLAY to ensure that ssh runs the program. **done**

Looking at ssh.exe, I think this will even work on Windows; it contains the
code to run ssh-askpass. (It does work on Windows!)

### securely handling the password

* Maybe force upgrade webapp to https? Locally, the risk would be that
  root could tcpdump and read password, so not large risk. If webapp
  is being accessed remotely, absolutely: require https.
* Use hs-securemem to store password.
* Avoid storing password for long. Erase it after webapp setup of remote
  is complete. Time out after 10 minutes and erase it. **done**
* If the user is slow, the cached ssh key can exire before they finish.
  This results in ssh being given no password, and failing. The UI
  now detects this and suggests the user retry. **done**
* Prompt using a html field name that does not trigger web browser password
  saving if possible.

### ssh-askpass shim, and password forwarding

`SSH_ASKPASS` needs to be set to a program (probably git-annex)
which gets the password from the webapp, and outputs it to stdout. **done**

Seems to call for the webapp and program to communicate over a local
socket (locked down so only user can access) or environment.
Environment is not as secure (easily snooped by root).
Local socket probably won't work on Windows. Could just use a temp file.

(Currently uses a temp file with locked down perms that it's careful
to clean up after use.)

Note that the webapp can probe to see if ssh needs a password, and can
prompt the user for it before running ssh and the ssh-askpass shim.
This avoids some complexity, and perhaps some attack vectors,
if the shim cannot requst an arbitrary password prompt.
(This complexity not needed with the temp file approach..)

### TODO

* test on OSX
* test on Android
* remove the vestigial terminal on Windows and Android, since this was the
  last thing actually using it (not easy!)


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnXybLxkPMYpP3yw4b_I6IdC3cKTD-xEdU"
 nickname="Matt"
 subject="comment 1"
 date="2014-04-29T10:10:04Z"
 content="""
Can you do something similar for gpg-agent?  Or do it instead with a ssh supported gpg-agent?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="216.145.95.162"
 subject="comment 2"
 date="2014-05-19T15:47:19Z"
 content="""
I don't feel comfortable prompting for gpg passphrases in the webapp. See [[gpgkeys]] for thoughts on this.
"""]]


code:
Once files are added (or removed or moved), need to send those changes to
all the other git clones, at both the git level and the key/value level.

## misc TODO

* Test MountWatcher on LXDE.
* Add a hook, so when there's a change to sync, a program can be run
   and do its own signaling.
* --debug will show often unnecessary work being done. Optimise.
* Configurablity, including only enabling git syncing but not data transfer;
  only uploading new files but not downloading, and only downloading
  files in some directories and not others. See for use cases:
  [[forum/Wishlist:_options_for_syncing_meta-data_and_data]]
* speed up git syncing by using the cached ssh connection for it too
  Will need to use `GIT_SSH`, which needs to point to a command to run,
  not a shell command line. Beware that the network connection may have
  bounced and the cached ssh connection not be usable.
* Map the network of git repos, and use that map to calculate
  optimal transfers to keep the data in sync. Currently a naive flood fill
  is done instead. Maybe use XMPP as a side channel to learn about the
  network topology?
* Find a more efficient way for the TransferScanner to find the transfers
  that need to be done to sync with a remote. Currently it walks the git
  working copy and checks each file. That probably needs to be done once,
  but further calls to the TransferScanner could eg, look at the delta
  between the last scan and the current one in the git-annex branch.
* [[use multiple transfer slots|todo/Slow_transfer_for_a_lot_of_small_files.]]
* The TransferQueue's list of deferred downloads could theoretically
  grow without bounds in memory. Limit it to a given number of entries,
  and fall back to some other method -- either storing deferred downloads
  on disk, or perhaps scheduling a TransferScanner run to get back into sync.

## More efficient syncing

See [[syncing/efficiency]]

## TransferScanner efficiency

The TransferScanner thread needs to find keys that need to be Uploaded
to a remote, or Downloaded from it.

How to find the keys to transfer? I'd like to avoid potentially
expensive traversals of the whole git working copy if I can.
(Currently, the TransferScanner does do the naive and possibly expensive
scan of the git working copy.)

One way would be to do a git diff between the (unmerged) git-annex branches
of the git repo, and its remote. Parse that for lines that add a key to
either, and queue transfers. That should work fairly efficiently when the
remote is a git repository. Indeed, git-annex already does such a diff
when it's doing a union merge of data into the git-annex branch. It
might even be possible to have the union merge and scan use the same
git diff data.

But that approach has several problems:

1. The list of keys it would generate wouldn't have associated git
   filenames, so the UI couldn't show the user what files were being
   transferred.
2. Worse, without filenames, any later features to exclude
   files/directories from being transferred wouldn't work.
3. Looking at a git diff of the git-annex branches would find keys
   that were added to either side while the two repos were disconnected.
   But if the two repos' keys were not fully in sync before they
   disconnected (which is quite possible; transfers could be incomplete),
   the diff would not show those older out of sync keys.

The remote could also be a special remote. In this case, I have to either
traverse the git working copy, or perhaps traverse the whole git-annex
branch (which would have the same problems with filesnames not being
available).

If a traversal is done, should check all remotes, not just
one. Probably worth handling the case where a remote is connected
while in the middle of such a scan, so part of the scan needs to be
redone to check it.

## done

1. Can use `git annex sync`, which already handles bidirectional syncing.
   When a change is committed, launch the part of `git annex sync` that pushes
   out changes. **done**; changes are pushed out to all remotes in parallel
1. Watch `.git/refs/remotes/` for changes (which would be pushed in from
   another node via `git annex sync`), and run the part of `git annex sync`
   that merges in received changes, and follow it by the part that pushes out
   changes (sending them to any other remotes).
   [The watching can be done with the existing inotify code! This avoids needing
   any special mechanism to notify a remote that it's been synced to.]
   **done**
1. Periodically retry pushes that failed.  **done** (every half an hour)
1. Also, detect if a push failed due to not being up-to-date, pull,
   and repush. **done**
2. Use a git merge driver that adds both conflicting files,
   so conflicts never break a sync. **done**

* on-disk transfers in progress information files (read/write/enumerate)
  **done**
* locking for the files, so redundant transfer races can be detected,
  and failed transfers noticed **done**
* transfer info for git-annex-shell **done**
* update files as transfers proceed. See [[progressbars]]
  (updating for downloads is easy; for uploads is hard)
* add Transfer queue TChan **done**
* add TransferInfo Map to DaemonStatus for tracking transfers in progress.
  **done**
* Poll transfer in progress info files for changes (use inotify again!
  wow! hammer, meet nail..), and update the TransferInfo Map **done**
* enqueue Transfers (Uploads) as new files are added to the annex by
  Watcher. **done**
* enqueue Tranferrs (Downloads) as new dangling symlinks are noticed by
  Watcher. **done**
  (Note: Needs git-annex branch to be merged before the tree is merged,
  so it knows where to download from. Checked and this is the case.)
* Write basic Transfer handling thread. Multiple such threads need to be
  able to be run at once. Each will need its own independant copy of the
  Annex state monad. **done**
* Write transfer control thread, which decides when to launch transfers.
  **done**
* Transfer watching has a race on kqueue systems, which makes finished
  fast transfers not be noticed by the TransferWatcher. Which in turn
  prevents the transfer slot being freed and any further transfers
  from happening. So, this approach is too fragile to rely on for
  maintaining the TransferSlots. Instead, need [[todo/assistant_threaded_runtime]],
  which would allow running something for sure when a transfer thread
  finishes. **done**
* Test MountWatcher on KDE, and add whatever dbus events KDE emits when
  drives are mounted. **done**
* It would be nice if, when a USB drive is connected,
  syncing starts automatically. Use dbus on Linux? **done**
* Optimisations in 5c3e14649ee7c404f86a1b82b648d896762cbbc2 temporarily
  broke content syncing in some situations, which need to be added back.
  **done**

  Now syncing a disconnected remote only starts a transfer scan if the
  remote's git-annex branch has diverged, which indicates it probably has
  new files. But that leaves open the cases where the local repo has
  new files; and where the two repos git branches are in sync, but the
  content transfers are lagging behind; and where the transfer scan has
  never been run.

  Need to track locally whether we're believed to be in sync with a remote.
  This includes:
  * All local content has been transferred to it successfully.
  * The remote has been scanned once for data to transfer from it, and all
    transfers initiated by that scan succeeded.

  Note the complication that, if it's initiated a transfer, our queued
  transfer will be thrown out as unnecessary. But if its transfer then
  fails, that needs to be noticed.

  If we're going to track failed transfers, we could just set a flag,
  and use that flag later to initiate a new transfer scan. We need a flag
  in any case, to ensure that a transfer scan is run for each new remote.
  The flag could be `.git/annex/transfer/scanned/uuid`.

  But, if failed transfers are tracked, we could also record them, in
  order to retry them later, without the scan. I'm thinking about a
  directory like `.git/annex/transfer/failed/{upload,download}/uuid/`,
  which failed transfer log files could be moved to.
* A remote may lose content it had before, so when requeuing
  a failed download, check the location log to see if the remote still has
  the content, and if not, queue a download from elsewhere. (And, a remote
  may get content we were uploading from elsewhere, so check the location
  log when queuing a failed Upload too.) **done**
* Fix MountWatcher to notice umounts and remounts of drives. **done**
* Run transfer scan on startup. **done**
* Often several remotes will be queued for full TransferScanner scans,
  and the scan does the same thing for each .. so it would be better to
  combine them into one scan in such a case. **done**
* The syncing code currently doesn't run for special remotes. While
  transfering the git info about special remotes could be a complication,
  if we assume that's synced between existing git remotes, it should be
  possible for them to do file transfers to/from special remotes.
  **done**

* The transfer code doesn't always manage to transfer file contents.

  Besides reconnection events, there are two places where transfers get queued:

  1. When the committer commits a file, it queues uploads.
  2. When the watcher sees a broken symlink be created, it queues downloads.

  Consider a doubly-linked chain of three repositories, A B and C.
  (C and A do not directly communicate.)

  * File is added to A.
  * A uploads its content to B.
  * At the same time, A git syncs to B.
  * Once B gets the git sync, it git syncs to C.
  * When C's watcher sees the file appear, it tries to download it. But if
    B had not finished receiving the file from A, C doesn't know B has it,
    and cannot download it from anywhere.

  Possible solution: After B receives content, it could queue uploads of it
  to all remotes that it doesn't know have it yet, which would include C.
  **done**

  In practice, this had the problem that when C receives the content,
  it will queue uploads of it, which can send back to B (or to some other repo
  that already has the content) and loop, until the git-annex branches catch
  up and break the cycle.

  To avoid that problem, incoming uploads should not result in a transfer
  info file being written when the key is already present. **done**

  Possible solution: C could record a deferred download. (Similar to a failed
  download, but with an unknown source.) When C next receives a git-annex
  branch push, it could try to queue deferred downloads. **done**

  Note that this solution won't cover use cases the other does. For example,
  connect a USB drive A; B syncs files from it, and then should pass them to C.
  If the files are not new, C won't immediatly request them from B.

* Running the assistant in a fresh clone of a repository, it sometimes
  skips downloading a file, while successfully downloading all the rest.
  There does not seem to be an error message. This will sometimes reproduce
  (in a fresh clone each time) several times in a row, but then stops happening,
  which has prevented me from debugging it.
  This could possibly have been caused by the bug fixed in 750c4ac6c282d14d19f79e0711f858367da145e4.
  Provisionally closed.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawk4YX0PWICfWGRLuncCPufMPDctT7KAYJA"
 nickname="betabrain"
 subject="selective data syncing"
 date="2012-07-24T15:27:08Z"
 content="""
How will the assistant know which files' data to distribute between the repos?

I'm using git-annex and it's numcopies attribute to maintain a redundant archive spread over different computers and usb drives. Not all drives should get a copy of everything, e.g. the usb drive I take to work should not automatically get a copy of family pictures.

How about .gitattributes?

* \* annex.auto-sync-data = false # don't automatically sync the data
* archive/ annex.auto-push-repos = NAS # everything added to archive/ in any repo goes automatically to the NAS remote.
* work/ annex.auto-synced-repos = LAPTOP WORKUSB # everything added to work/ in LAPTOP or WORKUSB gets synced to WORKUSB and LAPTOP
* work/ annex.auto-push-repos = LAPTOP WORKUSB # stuff added to work/ anywhere gets synced to LAPTOP and WORKUSB
* important/ annex.auto-sync-data = true # push data to all repos
* webserver_logs/ annex.remote.WEBSERVER.auto-push-repos = S3 # only the assistant running in WEBSERVER pushes webserver_logs/ to S3 remote
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnBl7cA6wLDxVNUyLIHvAyCkf8ir3alYpk"
 nickname="Tyson"
 subject="Bridging LANs"
 date="2012-07-10T10:20:59Z"
 content="""
Why rely on the cloud when you can instead use XMPP and jingle to perform NAT traversal for you? AFAIKT, it also means that traffic won't leave your router if the two endpoints are behind the same router.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 1"
 date="2012-07-03T08:26:43Z"
 content="""
On \"git syncing\" point number 9, on OSX you could potentially do this on a semi-regular basis

system_profiler SPNetworkVolumeDataType
Volumes:

    net:

      Type: autofs
      Mount Point: /net
      Mounted From: map -hosts
      Automounted: Yes

    home:

      Type: autofs
      Mount Point: /home
      Mounted From: map auto_home
      Automounted: Yes

and

x00:~ jtang$ system_profiler SPUSBDataType
USB:

    USB High-Speed Bus:

      Host Controller Location: Built-in USB
      Host Controller Driver: AppleUSBEHCI
      PCI Device ID: 0x0aa9
      PCI Revision ID: 0x00b1
      PCI Vendor ID: 0x10de
      Bus Number: 0x26

        Hub:

          Product ID: 0x2504
          Vendor ID: 0x0424  (SMSC)
          Version: 0.01
          Speed: Up to 480 Mb/sec
          Location ID: 0x26200000 / 3
          Current Available (mA): 500
          Current Required (mA): 2

            USB to ATA/ATAPI Bridge:

              Capacity: 750.16 GB (750,156,374,016 bytes)
              Removable Media: Yes
              Detachable Drive: Yes
              BSD Name: disk1
              Product ID: 0x2338
              Vendor ID: 0x152d  (JMicron Technology Corp.)
              Version: 1.00
              Serial Number: 313541813001
              Speed: Up to 480 Mb/sec
              Manufacturer: JMicron
              Location ID: 0x26240000 / 5
              Current Available (mA): 500
              Current Required (mA): 2
              Partition Map Type: MBR (Master Boot Record)
              S.M.A.R.T. status: Not Supported
              Volumes:
                Porta-Disk:
                  Capacity: 750.16 GB (750,156,341,760 bytes)
                  Available: 668.42 GB (668,424,208,384 bytes)
                  Writable: Yes
                  File System: ExFAT
....

I think its possible to programatically get this information either from the CLI (it dumps out XML output if required) or some development library. There is also DBUS in macports, but I have never had much interaction with it, so I don't know if its good or bad on OSX.
"""]]


code:
Currently, the git-annex assistant syncs with remotes in a way that is
dumb, and potentially inneficient:

1. Files are transferred to each reachable remote whose
   [[preferred_content]] setting indicates it wants the file.

2. After each file transfer (upload or download), a git sync
   is done to all the remotes, to update location log information.

## unncessary transfers

There are network toplogies where #1 is massively inneficient.
For example:

<pre>
  laptopA-----laptopB-----laptopC
      \         |             /
       \---cloud based repo--/
</pre>

When laptopA has a new file, it will first send it to laptopB. It will then
check if the cloud based transfer repository wants a copy. It will, because
laptopC has not yet gotten a copy. So laptopA will proceed with a slow
upload to the cloud, while meanwhile laptopB is sending the file over fast
LAN to laptopC.

(The more common case with no laptopC happens to work efficiently.
So does the case where laptopA is paired with laptopC.)

## unncessary syncing

Less importantly, the constant git syncing after each transfer is rather a
lot of work, and prevents collecting multiple presence changes to the git-annex
branch into larger commits, which would save disk space over time.

In many cases, this sync is necessary. For example, when a file is uploaded
to a transfer remote, the location change needs to be synced out so that
other clients know to grab it.

Or, when downloading a file from a drive, the sync lets other locally
paired repositories know we got it, so they can download it from us.
OTOH, this is also a case where a sync is sometimes unnecessary, since
if we're going to upload the file to them after getting it, the sync
only perhaps lets them start downloading it before our transfer queue
reaches a point where we'd upload it.

It would be good to find a way to detect when syncing is not immediately
necessary, and defer it.

## mapping

Mapping the repository network has the potential to get git-annex the
information it needs to avoid unnecessary transfers and/or unncessary
syncing.

Mapping the network can reuse code in `git annex map`. Once the map is
built, we want to find paths through the network that reach all nodes
eventually, with the least cost. This is a minimum spanning tree problem,
except with a directed graph, so really a Arborescence problem.

A significant problem in mapping is that nodes are mobile, they can move
between networks over time. This breaks LAN based paths through the
network. Mapping would need a way to detect this. Note that individual
git-annex assistants can tell when they've switched networks by using the
`networkConnectedNotifier`.

## P2P signaling

Another approach that might help with these problems is if git-annex
repositories have a non-git out of band signaling mechanism. This could,
for example, be used by laptopB to tell laptopA that it's trying to send
a file directly to laptopC. laptopA could then defer the upload to the
cloud for a while.

## syncing only requested content

See [[adhoc_routing]]


code:
[Telehash](http://telehash.org/) for secure P2P communication between
git-annex (assistant) repositories.

## telehash implementation status

* node.js version seems almost complete
* C version currently lacks channel support and seems buggy (13 Jan 2014)
* No pure haskell implementation of telehash v2. There was one of
  telehash v1 (even that seems incomplete). I have pinged its author
  to see if he anticipates updating it.
* Rapid development, situation may change in a month or 2.
* Is it secure? A security review should be done by competant people
  (not Joey). See <https://github.com/telehash/telehash.org/issues/23>
* **Haskell version**
  <https://github.com/alanz/htelehash/tree/v2/src/TeleHash>
  Development on v2 in haskell is just starting up!

## implementation basics

* Add a telehash.log that maps between uuid and telehash address.
* On startup, assistant creates a new telehash keypair if not already
  present; stores this locally and generates a telehash address from it,
  stored in telehash.log.
* Use telehash for notifications of changes to the repository
* Do git push over telehash. (Pretty easy, may need rate limiting in
  situations involving relays.)
* Remove git push over XMPP (which has several problems including
  XMPP being an unreliable transport, requiring a separate XMPP account per
  repo, and XMPP not being end-to-end encrypted)

## telehash address discovery

* Easy way is any set of repos that are already connected can communicate
  them via telehash.log.
* Local pairing can be used for telehash address discovery. Could be made
  to work without ssh (with content transfer over telehash discussed
  below).
* XMPP pairing can also be used for telehash address discovery. (Note that
  MITM attacks are possible.) Is it worth keeping XMPP in git-annex just
  for this?
* Telehash addresses of repositories can be communicated out of band (eg,
  via an OTR session or gpg signed mail), and pasted into the webapp to
  initiate a repository pairing that then proceeds entirely over telehash.
  Once both sides do this, the pairing can proceed automatically.

## content transfer over telehash

* In some circumstances, it would be ok to do annexed content transfer
  over telehash.
  Need to check if there are MTU problems with large data bodies in
  telehash messages.
  Probably not when a bridge is being used, due to required rate
  limiting in bridging over telehash. Cloud transfer remotes still needed for
  those situations.
* On a LAN, telehash can be used to determine the current local IP address
  of another computer on the LAN. The 2 could then determine if either uses
  ssh and if so use regular git-annex-shell for transfers. Or could do
  annexed content transfer directly over telehash.

## generic git-remote-telehash

This might turn out to be easy to split off from git-annex, so `git pull`
and `git push` can be used at the command line to access telehash remotes.
Allows using general git entirely decentralized and with end-to-end
encryption.

## separate daemon?

See [[git-remote-daemon]] for a design.

Advantages:

* `git annex sync` could also use the running daemon
* `git-remote-telehash` could use the running daemon
* c-telehash might end up linked to openssl, which has licence combination
  problems with git-annex. A separate process not using git-annex's code
  would avoid this.
* Allows the daemon to be written in some other language if necessary
  (for example, if c-telehash development stalls and the nodejs version is
  already usable)
* Potentially could be generalized to handle other similar protocols.
  Or even the xmpp code moved into it. There could even be git-annex native
  exchange protocols implemented in such a daemon to allow SSH-less
  transfers.
* Security holes in telehash would not need to compromise the entire
  git-annex. daemon could be sandboxed in one way or another.

Disadvantages:

* Adds some complexity.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlUbH3eytydcwlWqv8oauE2Jg4NwcV9uA0"
 nickname="Anna"
 subject="Special"
 date="2012-07-20T23:45:15Z"
 content="""
I feel pretty special getting an individualized thank you!  Btw, the good news is that your video finally explained what you were working on so that I understood it. :-)
"""]]


code:
This is a subset of [[/todo]] and [[/bugs]]
for items tagged for the assistant.

[[!inline pages="tagged(design/assistant) and !link(bugs/done) and !link(bugs/moreinfo)" show=0 archive=yes]]


code:
Some remotes are too small to sync everything to them.

The case of a small remote on a gadget that the user interacts with,
such as a phone, where they may want to request it get content
it doesn't currently have, is covered by the [[partial_content]] page.

But often the remote is just a removable drive or a cloud remote,
that has a limited size. This page is about making the assistant do
something smart with such remotes.

## TODO

* The expensive scan currently makes one pass, dropping content at the same
  time more uploads and downloads are queued. It would be better to drop as
  much content as possible upfront, to keep the total annex size as small
  as possible. How to do that without making two expensive scans?
* The TransferWatcher's finishedTransfer function relies on the location
  log having been updated after a transfer. But there's a race; if the
  log is not updated in time, it will fail to drop unwanted content.
  (There's a 10 second sleep there now to avoid the race, but that's hardly
  a fix.)

### dropping no longer preferred content

When a file is renamed, it might stop being preferred, so
could be checked and dropped. (If there's multiple links to
the same content, this gets tricky. Let's assume there are not.)

### analysis of changes that can result in content no longer being preferred

1. The preferred content expression can change, or a new repo is added, or
   groups change. Generally, some change to global annex state. Only way to deal
   with this is an expensive scan. (The rest of the items below come from
   analizing the terminals used in preferred content expressions.) **done**
2. renaming of a file (ie, moved to `archive/`) **done**
   (note also that renaming a file can also make it become preferred content
   again, and should cause it to be transferred in that case) **done**
3. we get a file (`in`, `copies`) **done**
4. we sent a file (`in`, `copies`) **done**
5. some other repository drops the file (`in`, `copies` .. However, it's
   unlikely that an expression would prefer content when *more* copies
   exisited, and want to drop it when less do. That's nearly a pathological
   case.)
6. `migrate` is used to change a backend (`inbackend`; unlikely)

That's all! Of these, 1-4 are by far the most important.

## specifying what data a remote prefers to contain (**done**)

Imagine a per-remote preferred content setting, that matches things that
should be stored on the remote.

For example, a MP3 player might use:
`smallerthan(10mb) and filename(*.mp3) and (not filename(junk/*))`

Adding that as a filter to files sent to a remote should be
straightforward.

A USB drive that is carried between three laptops and used to sync data
between them might use: `not (in=laptop1 and in=laptop2 and in=laptop3)`

In this case, transferring data from the usb repo should
check if preferred content settings rejects the data, and if so, drop it
from the repo. So once all three laptops have the data, it is
pruned from the transfer drive.

## repo groups (**done**)

Seems like git-annex needs a way to know the groups of repos. Some
groups:

* enduser: The user interacts with this repo directly.
* archival: This repo accumulates stuff, and once it's in enough archives,
  it tends to get removed from other places.
* transfer: This repo is used to transfer data between enduser repos,
  it does not hold data for long periods of time, and tends to have a
  limited size.

Add a group.log that can assign repos to these or other groups. (**done**)

Some examples of using groups:

* Want to remove content from a repo, if it's not an archival repo,
  and the content has reached at least one archival repo:

  `(not group=archival) and (not copies=archival:1)`

  That would make send to configure on all repos, or even set
  a global `annex.accept` to it. **done**

* Make a cloud repo only hold data until all known clients have a copy:

  `not ingroup(enduser)`

## configuration

The above is all well and good for those who enjoy boolean algebra, but
how to configure these sorts of expressions in the webapp?

Currently, we have a simple drop down list to select between a few
predefined groups with pre-defined preferred content recipes. Is this good
enough?

I think so; useful recipes can be developed on the wiki and included in
git-annex.

## the state change problem (**done**)

Imagine that a trusted repo has setting like `not copies=trusted:2`
This means that `git annex get --auto` should get files not in 2 trusted
repos. But once it has, the file is in 3 trusted repos, and so `git annex
drop --auto` should drop it again!

How to fix? Can it even be fixed? Maybe care has to be taken when
writing expressions, to avoid this problem. One that avoids it:
`not (copies=trusted:2 or (in=here and trusted=here and copies=trusted:3))`

Or, expressions could be automatically rewritten to avoid the problem.

Or, perhaps simulation could be used to detect the problem. Before
dropping, check the expression. Then simulate that the drop has happened.
Does the expression now make it want to add it? Then don't drop it!
**done**.. effectively using this approach.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 1"
 date="2012-09-23T21:58:08Z"
 content="""
You could try to do it similar to [RT](http://bestpractical.com/rt/):

* Implement saved statements, i.e. offer common use cases
* Allow those statements (or aliases? stanzas?) to be loaded in a relatively simple editor with a basic wizard to support the building of new rules
* Offer a free-text input for advanced users. It should be able to parse that and load it into the simple editor.
* Users should be able to save, export, and import those statements.
* Optionally, allow users to dry run the rules by showing them what git-annex needs to do to fulfill the requirements set by the statements.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnRRUUZo3W7pAoRoST8P_l0PtUBhvYuzDg"
 nickname="Lyle"
 subject="exactly this use case: cloud remotes too small"
 date="2014-01-06T00:31:34Z"
 content="""
I want to store a lot of scientific data in git-annex and have only a specific subset of my often used data in a box.com account so it is quickly synced between computers (my home internet has a limited upload speed). My home server will store all the files as it has effectively unlimited space.

I basically want the opposite of archive for git-annex-assistant, I'd like to mark the box.com special remote to store every *except* a certain special directory called less used files or whatever, or vise-versa, only store a folder of often used files, the local clients would store everything and my home server would also store everything.

It seems like transfer control and the groups.log can do this but I'm just not sure how to actually set it up?

Thanks!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.35"
 subject="comment 3"
 date="2014-01-06T15:04:33Z"
 content="""
Just put the box.com remote in the transfer group. There is no need for special subdirectories, the transfer group makes the remote only want files that have not yet reached all the known clients.
"""]]


code:
The assistant should support upgrading itself.

## non-distro upgrades

When git-annex was installed from this website, the assistant should poll
periodically (once a day or so) to see if there is a new version.
It downloads, over https, a .info file, which contains a serialized data
type containing upgrade information. The url it's downloaded from is
configured by setting `UPGRADE_LOCATION` when building git-annex on the
autobuilders.

When a new version is found, the webapp prompts the user to start the
upgrade. (annex.autoupgrade can be set to true to upgrade w/o prompting.)

The upgrade process is automatic, and rather tricky. The file is downloaded
using git-annex (as a regular key!), and is then unpacked into a new
directory, and the programfile updated to point to it. Then git-annex
restarts itself.

### manifest files

To clean up the old installation, a git-annex.MANIFEST file is looked for
in it, and the files listed, as well as empty directories, are deleted.
I don't want to accidentially delete something I didn't ship!

## restart on upgrade

When git-annex is installed from a proper distribution package, there is no
need for the above. But, the assistant still needs to notice when git-annex
get upgraded, and offer to restart (or automatically restart when
annex.autoupgrade is set).

This is done using the DirWatcher, watching the directory containing the
git-annex binary. Or, in the case of a non-distro install, watching the
directory where eg git-annex.linux/ was unpacked.

When an change is detected, restart.

## multi-daemon upgrades

A single system may have multiple assistant daemons running in different
repositories.

In this case, one daemon should do the non-distro upgrade, and the rest
should notice the upgrade and restart.

I don't want every daemon trying to download the file at once..

Approach: The first new version is installed into a stable directory, based
on its version. So, start the upgrade by making this directory. If upgrade
is already in progress, the directory will already exist. (Remove directory
if upgrade fails.)


code:
The webapp is a web server that displays a shiny interface.

## performance

Having the webapp open while transfers are
running uses significant CPU just for the browser to update the progress
bar. Unsurprising, since the webapp is sending the browser a new `<div>`
each time. Updating the DOM instead from javascript would avoid that;
the webapp just needs to send the javascript either a full `<div>` or a
changed percentage and quantity complete to update a single progress bar.

(Another reason to do this is it'll cut down on the refreshes, which
sometimes make browsers ignore clicks on UI elements like the pause button,
if the transfer display refreshes just as the click is made.)

## other features

* there could be a UI to export a file, which would make it be served up
  over http by the web app
* there could be a UI (some javascript thing) in the web browser to
  submit urls to the web app to be added to the annex and downloaded.
  See: [[todo/wishlist:_an_"assistant"_for_web-browsing_--_tracking_the_sources_of_the_downloads]]
* Display the `inotify max_user_watches` exceeded message. **done**
* Display something sane when kqueue runs out of file descriptors.
* allow removing git remotes **done**
* allow disabling syncing to here, which should temporarily disable all
  local syncing. **done**

## better headless support

`--listen` is insecure, and using HTTPS would still not make it 100% secure
as there would be no way for the browser to verify its certificate.

I do have a better idea, but it'd be hard to implement.
`git annex webapp --remote user@host:dir` could ssh to the remote host,
run the webapp there, listening only on localhost, and then send the
port the webapp chose back over the ssh connection. Then the same
ssh connection could be reused (using ssh connection caching) to set up
port forwarding from a port on the local host to the remote webapp.

This would need to handle the first run case too, which would require
forwarding a second port once the webapp made the repository and
the second webapp started up.

## first start **done**

* make git repo **done**
* generate a nice description like "joey@hostname Desktop/annex" **done**
* record repository that was made, and use it next time run **done**
* write a pid file, to prevent more than one first-start process running
  at once **done**

## security **acceptable/done**

* Listen only to localhost. **done**
* Instruct the user's web browser to open an url that contains a secret
  token. This guards against other users on the same system. **done**
  (I would like to avoid passwords or other authentication methods,
  it's your local system.)
* Don't pass the url with secret token directly to the web browser,
  as that exposes it to `ps`. Instead, write a html file only the user can read,
  that redirects to the webapp. **done**
* Alternative for Linux at least would be to write a small program using
  GTK+ Webkit, that runs the webapp, and can know what user ran it, avoiding
  needing authentication.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlup4hyZo4eCjF8T85vfRXMKBxGj9bMdl0"
 nickname="Ben"
 subject="ARM support"
 date="2012-07-13T16:51:15Z"
 content="""
The closure of [this](http://hackage.haskell.org/trac/ghc/ticket/5839) ticket hopefully marks the end of TH issues on ARM. As of 7.4.2, GHC's linker has enough ARM support to allow a selection of common packages compile on my PandaBoard. That being said, it hasn't had a whole lot of testing so it's possible I still need to implement a few relocation types.
"""]]


code:
[[!comment format=mdwn
 username="yatesa"
 ip="171.25.193.21"
 subject="Secret URL token"
 date="2012-06-19T03:41:16Z"
 content="""
> Instruct the user's web browser to open an url that contains a secret token. This guards against other users on the same system.

How will you implement that? Running \"sensible-browser URL\" would be the obvious way, but the secret URL would show up in a well timed ps listing. (And depending on the browser, ps may show the URL the entire time it's running.)
"""]]


code:
[[!comment format=mdwn
 username="jtang"
 ip="79.97.135.214"
 subject="comment 3"
 date="2012-07-26T17:35:18Z"
 content="""
Using twitter-bootstrap for the webapp - this might be a wishlist item, but would it be possible to ensure that the webapp's css uses twitter-bootstrap classes. It would make theming much easier in the long run and it would give you a nice modern look with a low amount of effort.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.2.189"
 subject="comment 4"
 date="2012-07-26T17:45:28Z"
 content="""
So, Yesod's scaffolded site actually does use bootstrap, but I didn't use the scaffolded site so don't have it. I am not quite to the point of doing any theming of the webapp, but I do have this nice example of how to put in bootstrap right here..

By the way, if anyone would like to play with the html templates for the webapp, the main html template is `templates/default-layout.hamlet`. Uses a slightly weird template markup, but plain html will also work. And there's also the `static/` directory; every file in there will be compiled directly into the git-annex binary, and is available at `http://localhost:port/static/$file` in the webapp. See the favicon link in `default-layout.hamlet` of how to construct a type-safe link to a static file: `href=@{StaticR favicon_ico}`. That's all you really need to theme the webapp, without doing any real programming!
"""]]


code:
See [[todo/windows_support]]..

## symlinks

Apparently new versions of Windows have something very like symlinks.
(Or really, 3 or so things not entirely unlike symlinks and all different.)
Stackoverflow has some details.

NTFS supports symbolic links two different ways: an [[!wikipedia NTFS symbolic link]] and an [[!wikipedia NTFS_junction_point]].  The former seems like the closest analogue to POSIX symlinks.

The windows port will not use symlinks. It will only support direct mode.

## POSIX

Lots of ifdefs and pain to deal with POSIX calls in the code base.

Or I could try to use Cygwin.

## Deeper system integration

[NTFS Reparse Points](http://msdn.microsoft.com/en-us/library/aa365503%28v=VS.85%29.aspx) allow a program to define how the OS will interpret a file or directory in arbitrary ways.  This requires writing a file system filter.

## Developement environment

Someone wrote in to say:

> For Windows Development you can easily qualify
> for Bizspark - http://www.microsoft.com/bizspark/
>
> This will get you 100% free Windows OS licenses and
> Dev tools, plus a free Azure account for cloud testing.
> (You can also now deploy Linux VMs to Azure as well)
> No money required at all.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlCGROoy62svBUy6P24x1KoGoDWrBq2ErA"
 nickname="Steve"
 subject="comment 1"
 date="2012-08-07T04:15:43Z"
 content="""
NTFS symbolic links should do exactly what you would expect them to do. They can point to files or directories. Junction points are legacy NTFS functionality and reparse points are more like the POSIX mount functionality.

NTFS symbolic links should work for you, junction point should be avoided, and reparse points would be like using a nuke to kill a fly. The only hang up you might have is that I think all three features require elevated privileges to manage.
"""]]


code:
The git-annex assistant uses XMPP to communicate between peers that
cannot directly talk to one-another. A typical scenario is two users
who share a repository, that is stored in the [[cloud]].

### TODO

* Do git-annex clients sharing an account with regular clients cause confusing
  things to happen?
  See <http://git-annex.branchable.com/design/assistant/blog/day_114__xmpp/#comment-aaba579f92cb452caf26ac53071a6788>
* Support registering with XMPP provider from within the webapp,
  as clients like pidgin are able to do.
* At least some XMPP servers are lossy (does XMPP guarantee delivery)?
  I have seen a log where a push's packet 1 and 3 arrived, but 2 did not.
  To deal with this, need at least a 1 packet buffer and ACK or resend
  request implemented over top of XMPP. Essentially, TCP over XMPP. :(

## design goals

1. Avoid user-visible messages. dvcs-autosync uses XMPP similarly, but
   sends user-visible messages. Avoiding user-visible messages lets
   the user configure git-annex to use his existing XMPP account
   (eg, Google Talk).

2. Send notifications to buddies. dvcs-autosync sends only self-messages,
   but that requires every node have the same XMPP account configured.
   git-annex should support that mode, but it should also send notifications
   to a user's buddies. (This will also allow for using XMPP for pairing
   in the future.)

3. Don't make account appear active. Just because git-annex is being an XMPP
   client, it doesn't mean that it wants to get chat messages, or make the
   user appear active when he's not using his chat program.

## protocol

To avoid relying on XMPP extensions, git-annex communicates
using presence messages, and chat messages (with empty body tags,
so clients don't display them).

git-annex sets a negative presence priority, to avoid any regular messages
getting eaten by its clients. It also sets itself extended away.
Note that this means that chat messages always have to be directed at
specific git-annex clients.

To the presence and chat messages, it adds its own tag as
[extended content](http://xmpp.org/rfcs/rfc6121.html#presence-extended).
The xml namespace is "git-annex" (not an URL because I hate wasting bandwidth).

To indicate it's pushed changes to a git repo with a given UUID, a message
that is sent to all buddies and other clients using the account (no
explicit pairing needed), it uses a broadcast presence message containing:

	<git-annex xmlns='git-annex' push="uuid[,uuid...]" />

Multiple UUIDs can be listed when multiple clients were pushed. If the
git repo does not have a git-annex UUID, an empty string is used.

To query if other git-annex clients are around, a presence message is used,
containing:

	<git-annex xmlns='git-annex' query="" />

For pairing, a chat message is sent to every known git-annex client,
containing:

	<git-annex xmlns='git-annex' pairing="PairReq|PairAck|PairDone myuuid" />

### git push over XMPP

To indicate that we could push over XMPP, a chat message is sent,
to each known client of each XMPP remote.

	<git-annex xmlns='git-annex' canpush="myuuid" shas="sha1 sha1" />

The shas are omitted by old clients. If present, they are the git shas of
the head and git-annex branches that are available to be pushed. This lets
the receiver check if it's already got them.

To request that a remote push to us, a chat message can be sent.

	<git-annex xmlns='git-annex' pushrequest="myuuid" />

When replying to an canpush message, this is directed at the specific
client that indicated it could push. To solicit pushes from all clients,
the message has to be sent directed individually to each client.

When a peer is ready to send a git push, it sends:

	<git-annex xmlns='git-annex' startingpush="myuuid" />

The receiver runs `git receive-pack`, and sends back its output in
one or more chat messages, directed to the client that is pushing:

	<git-annex xmlns='git-annex' rp="N">
	007b27ca394d26a05d9b6beefa1b07da456caa2157d7 refs/heads/git-annex report-status delete-refs side-band-64k quiet ofs-delta
	</git-annex>

The sender replies with the data from `git push`, in
one or more chat messages, directed to the receiver:

	<git-annex xmlns='git-annex' sp="N">
	data
	</git-annex>

The value of rp and sp used to be empty, but now it's a sequence number.
This indicates the sequence of this packet, counting from 1. The receiver
and sender each have their own sequence numbers.

When `git receive-pack` exits, the receiver indicates its exit
status with a chat message, directed at the sender:

	<git-annex xmlns='git-annex' rpdone="0" />

### security

Data git-annex sends over XMPP will be visible to the XMPP account's
buddies, and to the XMPP server (and any attacker who has access to the
XMPP server). So it's important to consider the security exposure of using
it.

Even if git-annex sends only a single bit notification, this lets attackers
know when the user is active and changing files. Although the assistant's other
syncing activities can somewhat mask this.

As soon as git-annex does anything unlike any other client, an attacker can
see how many clients are connected for a user, and fingerprint the ones
running git-annex, and determine how many clients are running git-annex.

An attacker can observe the UUIDs used for pushes and pairing, and determine
how many different remotes are being used.

An attacker could replay push notification messages, reusing UUIDs it's
observed. This would make clients pull repeatedly, perhaps as a DOS.

The XMPP server, or an attacker with access to it can reconstruct the git
repository from data sent in pushes, in part or in whole.


code:
[[!comment format=mdwn
 username="http://meep.pl/"
 ip="193.23.174.18"
 subject="xmlns"
 date="2012-11-11T09:00:01Z"
 content="""
A minor point, but is saving a couple of bytes per message worth using a [deprecated feature](http://www.w3.org/TR/REC-xml-names/#iri-use) of the namespaces specification? This is not technically *breaking* the current specification, since \"git-annex\" is of course still a (relative) URI reference; and anyway chances of problems are, I guess, slim. But is it the lesser of two bugs?

The shortest moderately sane absolute URI containing \"git-annex\" would probably be \"data:,git-annex\".
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkurjhi0CRJvgm7QNaZDWS9hitBtavqIpc"
 nickname="Bret"
 subject="Plans for two factor authentication or oath?"
 date="2013-04-15T00:58:04Z"
 content="""
Are there plans to support google's two factor authentication?  Right now I have to use an application specific password.
"""]]


code:
[[!comment format=mdwn
 username="hhm"
 ip="108.17.80.177"
 subject="file transfer?"
 date="2013-04-23T10:22:51Z"
 content="""
Would it be possible to add optional support for transferring files over XMPP (possibly being disabled out-of-the-box so as not to suck up third-party bandwidth)?
"""]]


code:
Currently [[xmpp]] relies on the SSL connection to the server for security.
The server can see git repository data pushed through it. (Also, the SSL
connection is not pinned or really checked well at all.)

Add an encryption layer that does not rely on trusting the XMPP server's
security. There are a few options for how to generate the key for eg,
AES encryption:

  * Do a simple Diffie-Hellman shared key generation when starting each XMPP
    push session. Would not protect the users from active MITM by the
    XMPP server, but would prevent passive data gathering attacks from
    getting useful data. Since the key is ephemeral, would provide
    Forward Security.
  * Do D-H key generation, but at pairing, not push time. Allows for an
    optional confirmation step, using eg, BubbleBabble to compare the
    keys out of band. ("I see xebeb-dibyb-gycub-kacyb-modib-pudub-sefab-vifuc-bygoc-daguc-gohec-kuxax .. do you too?")
  * Prompt both users for a passphrase when XMPP pairing, and
    use SPEKE (or similar methods like J-PAKE) to generate a shared key.
    Avoids active MITM attacks. Makes pairing harder, especially pairing
    between one's own devices, since the passphrase has to be entered on
    all devices. Also problimatic when pairing more than 2 devices,
    especially when adding a device to the set later, since there
    would then be multiple different keys in use.
  * Rely on the user's gpg key, and do gpg key verification during XMPP
    pairing. Problimatic because who wants to put their gpg key on their
    phone? Also, require the users be in the WOT and be gpg literate.

Update: This seems unlikely to be worth doing. [[Telehash]] is better.
--[[Joey]]


code:
[[!comment format=mdwn
 username="Sagi"
 ip="2001:610:1908:8001:25ce:8a0b:53cc:c010"
 subject="Why not use OTR?"
 date="2013-07-23T23:53:12Z"
 content="""
[OTR](http://www.cypherpunks.ca/otr/) seems a natural fit for end-to-end encryption over xmpp. I can't find Haskell bindings for libotr at this moment, but creating those instead of inventing new protocols might be worth considering. There are also several [native implementations](http://www.cypherpunks.ca/otr/software.php) (Python, Go, Scheme), but I'm not sure if that's worth the effort.

"""]]


code:
* [[metadata]] for views
* [direct mode mappings scale badly with thousands of identical files](/bugs/__34__Adding_4923_files__34___is_really_slow)
* [[bugs/incremental_fsck_should_not_use_sticky_bit]]
* [[todo/wishlist:_pack_metadata_in_direct_mode]]
* [[todo/cache_key_info]]

What do all these have in common? They could all be improved by
using some kind of database to locally store the information in an
efficient way.

The database should only function as a cache. It should be able to be
generated and updated by looking at the git repository.

* Metadata can be updated by looking at the git-annex branch,
  either its current state, or the diff between the old and new versions
* Direct mode mappings can be updated by looking at the current branch,
  to see which files map to which key. Or the diff between the old
  and new versions of the branch.
* Incremental fsck information is not stored in git, but can be
  "regenerated" by running fsck again.
  (Perhaps doesn't quite fit, but let it slide..)

Store in the database the Ref of the branch that was used to construct it.
(Update in same transaction as cached data.)

## implementation plan

1. Implement for metadata, on a branch, with sqlite.
2. Make sure that builds on all platforms.
3. Add associated file mappings support. This is needed to fully
   use the caching database to construct views.
4. Store incremental fsck info in db.
5. Replace .map files with 3. for direct mode.

## case study: persistent with sqllite

Here's a non-normalized database schema in persistent's syntax.

CachedKey
  key Key
  associatedFiles [FilePath]
  lastFscked Int Maybe
  KeyIndex key

CachedMetaData
  key Key
  metaDataField MetaDataField
  metaDataValue MetaDataValue

Using the above database schema and persistent with sqlite, I made
a database containing 30k Cache records. This took 5 seconds to create
and was 7 mb on disk. (Would be rather smaller, if a more packed Key
show/read instance were used.)

Running 1000 separate queries to get 1000 CachedKeys took 0.688s with warm
cache. This was more than halved when all 1000 queries were done inside the
same `runSqlite` call. (Which could be done using a separate thread and some
MVars.)

(Note that if the database is a cache, there is no need to perform migrations
when querying it. My benchmarks skip `runMigration`. Instead, if the query
fails, the database doesn't exist, or uses an incompatable schema, and the
cache can be rebuilt then. This avoids the problem that persistent's migrations
can sometimes fail.)

Doubling the db to 60k scaled linearly in disk and cpu and did not affect
query time.

----

Here's a normalized schema:

CachedKey
  key Key
  KeyIndex key
  deriving Show

AssociatedFiles
  keyId CachedKeyId Eq
  associatedFile FilePath
  KeyIdIndex keyId associatedFile
  deriving Show

CachedMetaField
  field MetaField
  FieldIndex field

CachedMetaData
  keyId CachedKeyId Eq
  fieldId CachedMetaFieldId Eq
  metaValue String

LastFscked
  keyId CachedKeyId Eq
  localFscked Int Maybe

With this, running 1000 joins to get the associated files of 1000
Keys took 5.6s with warm cache. (When done in the same `runSqlite` call.) Ouch!

Update: This performance was fixed by adding `KeyIdOutdex keyId associatedFile`,
which adds a uniqueness constraint on the tuple of key and associatedFile.
With this, 1000 queries takes 0.406s. Note that persistent is probably not
actually doing a join at the SQL level, so this could be sped up using
eg, esquelito.

Update2: Using esquelito to do a join got this down to 0.250s.

Code: <http://lpaste.net/101141> <http://lpaste.net/101142>

Compare the above with 1000 calls to `associatedFiles`, which is approximately
as fast as just opening and reading 1000 files, so will take well under
0.05s with a **cold** cache.

So, we're looking at nearly an order of magnitude slowdown using sqlite and
persistent for associated files. OTOH, the normalized schema should
perform better when adding an associated file to a key that already has many.

For metadata, the story is much nicer. Querying for 30000 keys that all
have a particular tag in their metadata takes 0.65s. So fast enough to be
used in views.


code:
This was the design doc for [[/encryption]] and is preserved for
the curious. For an example of using git-annex with an encrypted S3 remote,
see [[tips/using_Amazon_S3]].

[[!toc]]

## encryption backends

It makes sense to support multiple encryption backends. So, there
should be a way to tell what backend is responsible for a given filename
in an encrypted remote. (And since special remotes can also store files
unencrypted, differentiate from those as well.)

The rest of this page will describe a single encryption backend using GPG.
Probably only one will be needed, but who knows? Maybe that backend will
turn out badly designed, or some other encryptor needed. Designing
with more than one encryption backend in mind helps future-proofing.

## encryption key management

[[!template id=note text="""
The basis of this scheme was originally developed by Lars Wirzenius et al
[for Obnam](http://liw.fi/obnam/encryption/).
"""]]

Data is encrypted by GnuPG, using a symmetric cipher. The cipher is
generated by GnuPG when the special remote is created. By default the
best entropy pool is used, hence the generation may take a while; One
can use `initremote` with `highRandomQuality=false` or `--fast` options
to speed up things, but at the expense of using random numbers of a
lower quality. The generated cipher is then checked into your git
repository, encrypted using one or more OpenPGP public keys. This scheme
allows new OpenPGP private keys to be given access to content that has
already been stored in the remote.

Different encrypted remotes need to be able to each use different ciphers.
Allowing multiple ciphers to be used within a single remote would add a lot
of complexity, so is not planned to be supported.
Instead, if you want a new cipher, create a new S3 bucket, or whatever.
There does not seem to be much benefit to using the same cipher for
two different encrypted remotes.

So, the encrypted cipher could just be stored with the rest of a remote's
configuration in `remotes.log` (see [[internals]]). When `git
annex intiremote` makes a remote, it can generate a random symmetric
cipher, and encrypt it with the specified gpg key. To allow another gpg
public key access, update the encrypted cipher to be encrypted to both gpg
keys.

## filename enumeration

If the names of files are encrypted or securely hashed, or whatever is
chosen, this makes it harder for git-annex (let alone untrusted third parties!)
to get a list of the files that are stored on a given enrypted remote.
But, does git-annex really ever need to do such an enumeration?

Apparently not. `git annex unused --from remote` can now check for
unused data that is stored on a remote, and it does so based only on
location log data for the remote. This assumes that the location log is
kept accurately.

What about `git annex fsck --from remote`? Such a command should be able to,
for each file in the repository, contact the encrypted remote to check
if it has the file. This can be done without enumeration, although it will
mean running gpg once per file fscked, to get the encrypted filename.

So, the files stored in the remote should be encrypted. But, it needs to
be a repeatable encryption, so they cannot just be gpg encrypted, that
would yeild a new name each time. Instead, HMAC is used. Any hash could
be used with HMAC. SHA-1 is the default, but [[other_hashes|/encryption]]
can be chosen for new remotes.

It was suggested that it might not be wise to use the same cipher for both
gpg and HMAC. Being paranoid, it's best not to tie the security of one
to the security of the other. So, the encrypted cipher described above is
actually split in two; half is used for HMAC, and half for gpg.

----

Does the HMAC cipher need to be gpg encrypted? Imagine if it were
stored in plainext in the git repository. Anyone who can access
the git repository already knows the actual filenames, and typically also
the content hashes of annexed content. Having access to the HMAC cipher
could perhaps be said to only let them verify that data they already
know.

While this seems a pretty persuasive argument, I'm not 100% convinced, and
anyway, most times that the HMAC cipher is needed, the gpg cipher is also
needed. Keeping the HMAC cipher encrypted does slow down two things:
dropping content from encrypted remotes, and checking if encrypted remotes
really have content. If it's later determined to be safe to not encrypt the
HMAC cipher, the current design allows changing that, even for existing
remotes.

## other use of the symmetric cipher

The symmetric cipher can be used to encrypt other content than the content
sent to the remote. In particular, it may make sense to encrypt whatever
access keys are used by the special remote with the cipher, and store that
in remotes.log. This way anyone whose gpg key has been given access to
the cipher can get access to whatever other credentials are needed to
use the special remote.

## risks

A risk of this scheme is that, once the symmetric cipher has been
obtained, it allows full access to all the encrypted content. Indeed
anyone owning a key that used to be granted access could already have
decrypted the cipher and stored a copy. While it is in possible to
remove a key with `keyid-=`, it is designed for a
[[completely_different_purpose|/encryption]] and does not actually revoke
access.

If git-annex stores the decrypted symmetric cipher in memory, then there
is a risk that it could be intercepted from there by an attacker. Gpg
ameliorates these type of risks by using locked memory. For git-annex, note
that an attacker with local machine access can tell at least all the
filenames and metadata of files stored in the encrypted remote anyway,
and can access whatever content is stored locally.

This design does not support obfuscating the size of files by chunking
them, as that would have added a lot of complexity, for dubious benefits.
If the untrusted party running the encrypted remote wants to know file sizes,
they could correlate chunks that are accessed together. Encrypting data
changes the original file size enough to avoid it being used as a direct
fingerprint at least.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 1"
 date="2011-04-03T20:03:14Z"
 content="""
New encryption keys could be used for different directories/files/patterns/times/whatever. One could then encrypt this new key for the public keys of other people/machines and push them out along with the actual data. This would allow some level of access restriction or future revocation. git-annex would need to keep track of which files can be decrypted with which keys. I am undecided if that information needs to be encrypted or not.

Encrypted object files should be checksummed in encrypted form so that it's possible to verify integrity without knowing any keys. Same goes for encrypted keys, etc.

Chunking files in this context seems like needless overkill. This might make sense to store a DVD image on CDs or similar, at some point. But not for encryption, imo. Coming up with sane chunk sizes for all use cases is literally impossible and as you pointed out, correlation by the remote admin is trivial.
"""]]