code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkurjhi0CRJvgm7QNaZDWS9hitBtavqIpc"
 nickname="Bret"
 subject="A few OS X bugs"
 date="2012-09-25T00:50:56Z"
 content="""
Cabal update

Cabal install git-annex
Seemed to install correctly on OS X 10.7

After the installation, an application for git-annex showed up on my desktop.  This is the wrong place to put applications in OS X.  It should belong in the /Applications Folder.
Launching the git-annex app, the web-browser is opened, and it tells me to make a repository.  I entered /Users/[my home folder]/Annex and continued.  I then got an internal server error and nothing else happened.  The Error:

uuid [\"-m\"] exited 127

Annex was created, and there is a .git folder in it.

Checking the annex status:
Bret-Air:Annex bret$ git annex status
git-annex: First run: git-annex init

I'll try it on 10.6 here in just a moment.  Let me know if I can provide any more useful information.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkurjhi0CRJvgm7QNaZDWS9hitBtavqIpc"
 nickname="Bret"
 subject="comment 4"
 date="2012-09-25T00:59:51Z"
 content="""
OS X 10.6 is doing the same thing as on 10.7.
"""]]

repo_name: kakafarm/git

license: unknown


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.14.141"
 subject="comment 5"
 date="2012-09-25T14:49:31Z"
 content="""
@Bret my best guess (without any error message from the uuid command) is that your uuid doesn't support the -m parameter. I've made the configure step check for this and it'll fall back to uuid without the parameter.
"""]]


code:
Mostly took a break from working on the assistant today. Instead worked
on adding incremental fsck to git-annex. Well, that will be something
that assistant will use, eventually, probably.

Jimmy and I have been working on a self-contained OSX app for using the
assistant, that doesn't depend on installing git, etc. More on that
once we have something that works.


code:
Amazon S3 was the second most popular choice in the
[[polls/prioritizing_special_remotes]] poll, and since I'm not sure how
I want to support phone/mp3 players, I did it first.

So I added a configurator today to easily set up an Amazon S3 repository.
That was straightforward and didn't take long since git-annex already
supported S3.

The hard part, of course, is key distribution. Since the webapp so far
can only configure the shared encryption method, and not fullblown gpg keys,
I didn't feel it would be secure to store the S3 keys in the git repository.
Anyone with access to that git repo would have full access to S3 ... just not
acceptable. Instead, the webapp stores the keys in a 600 mode file locally,
and they're not distributed at all.

When the same S3 repository is enabled on another computer, it prompts for
keys then too. I did add a hint about using the IAM Management Console in
this case -- it should be possible to set up users in IAM who can only
access a single bucket, although I have not tried to set that up.

---

Also, more work on the standalone OSX app.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 1"
 date="2012-09-26T23:52:50Z"
 content="""
I think a lot of people misinterpreted \"phone/mp3 player\" as \"Android\"; I know I did.

As you already support directories, MTP would be needed to support the other half of pretty much all phones and media players (other than those made by Apple). MTP is a valley of pain, though.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlFd2EbAmGD0FjHmuoerXfT0GO_jPcgvQA"
 nickname="Gert"
 subject="Single bucket for S3"
 date="2012-09-27T09:17:24Z"
 content="""
You can easily do that with IAM, checkout <http://andrewhitchcock.org/?post=325>.

It boils down to creating a policy for each user/bucket. But allow read access to the full bucket list (otherwise the AWS Console won't work).
"""]]


code:
Various bug fixes, and work on the OSX app today:

* Avoid crashing when ssh-keygen fails due to not being able to parse
  `authorized_keys`.. seems a lot of people have crufty unparsable
  `authorized_keys` files.
* On OSX, for some reason the webapp was failing to start sometimes due
  to bind failing with EINVAL. I don't understand why, as that should
  only happen if the socket is already bound, which it should not as
  it's just been created. I was able to work around this by retrying
  with a new socket when bind fails.
* When setting up `authorized_keys` to let `git-annex-shell` be run,
  it had been inserting a perl oneliner into it. I changed that
  to instead call a `~/.ssh/git-annex-shell` wrapper script that it sets
  up. The benefits are it no longer needs perl, and it's less ugly,
  and the standalone OSX app can modify the wrapper script to point to
  wherever it's installed today (people like to move these things around I
  guess).
* Made the standalone OSX app set up autostarting when it's first run.
* Spent rather a long time collecting the licenses of all the software that
  will be bundled with the standalone OSX app. Ended up with a file
  containing 3954 lines of legalese. Happily, all the software appears
  redistributable, and free software; even the couple of OSX system libraries
  we're bundling are licensed under the APSL.


code:
I hear that people want the git-annex assistant to be easy to install
without messing about building it from source..

## on OSX

So Jimmy and I have been working all week on making an easily installed OSX
app of the assistant. This is a .dmz file that bundles all the dependencies
(git, etc) in, so it can be installed with one click.

It seems to basically work. You can get it [[here|install/OSX]].

Unfortunatly, the [[bugs/pasting_into_annex_on_OSX]] bug resurfaced while
testing this.. So I can't really recommend using it on real data yet.

Still, any testing you can do is gonna be really helpful. I'm squashing OSX
bugs right and left.

## on Linux

First of all, the git-annex assistant is now available in Debian unstable,
and in Arch Linux's AUR. Proper packages.

For all the other Linux distributions, I have a workaround. It's
a big hack, but it seems to work.. at least on Debian stable.

I've just put up a [[install/linux_standalone]] tarball, which has **no
library dependencies** apart from glibc, and doesn't even need git to be
installed on your system.

## on FreeBSD

The FreeBSD port has been updated to include the git-annex assistant too..

[[!meta title="day 94 easy install"]]


code:
[[!comment format=mdwn
 username="http://rjc.pip.verisignlabs.com/"
 ip="86.22.66.200"
 subject="assistant in Debian"
 date="2012-09-30T09:10:17Z"
 content="""
The same version of ghc in Debian repository on i386 and amd64 privides ghc-ghci virtual package while on all the other architectures it doesn't. Is that normal or shall I file a bug?

What it basically means is that the packages, both git-annex (with assistant) and its build dependencies, for all the other archs cannot be build on Debian porting machines until that is fixed. It already is in the experimental version but until that one gets to unstable there won't be any official debs for all non-Intel architecures.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.149"
 subject="comment 2"
 date="2012-09-30T17:11:13Z"
 content="""
The next upload to debian will enable building the assistant on architectures that don't have template haskell and so cannot build the webapp.
"""]]


code:
Spent a lot of time this weekend thinking about/stuck on the [[cloud]]
notification problem. Currently IRC is looking like the best way for
repositories to notify one-another when changes are made, but I'm not sure
about using that, and not ready to start on it.

Instead, laid some groundwork for [[transfer_control]] today. Added
some simple commands to manage groups of repositories, and find files
that are present in repositories in a group. I'm not completely happy
with the syntax for that, and need to think up some good syntax to specify
files that are present in *all* repositories in a group.

The plan is to have the assistant automatically guess at groups to put new
repositories it makes in (it should be able to make good guesses),
as well as have an interface to change them, and an interface to configure
transfer control using these groups (and other ways of matching files).
And, probably, some canned transfer control recipes for common setups.

---

Collected up the past week's work and made a release today. I'm probably
back to making regular releases every week or two.


code:
Today I revisited something from way back in [[day_7__bugfixes]].
Back then, it wasn't practical to run `git ls-files` on every
file the watcher noticed, to check if it was already in git. Revisiting
this, I found I could efficiently do that check at the same point it checks
`lsof`. When there's a lot of files being added, they're batched up at that
point, so it won't be calling `git ls-files` repeatedly.

Result: It's safe to mix use of the assistant with files stored in git
in the normal way. And it's safe to mix use of `git annex unlock` with
the assistant; it won't immediately re-lock files. Yay!

----

Also fixed a crash in the committer, and made `git annex status` display
repository groups.

----

Been thinking through where to store the [[transfer_control]] expressions.
Since repositories need to know about the transfer controls of other
remotes, storing them in `.git/config` isn't right. I thought it might be
nice to configure the expressions in `.gitattributes`, but it seems the
file format doesn't allow complicated multi-word attributes. Instead,
they'll be stored in the git-annex branch.


code:
[[!comment format=mdwn
 username="http://wiggy.net/"
 nickname="Wichert"
 subject="symlinks?"
 date="2012-10-03T07:15:21Z"
 content="""
Does that mean that git-annex will no longer need to turn everything into symlinks? I suspect that that will improve behaviour on at least OSX quite a bit.
"""]]


code:
Not a lot of programming today; I spent most of the day stuffing hundreds
of envelopes for this Kickstarter thing you may have heard of. Some post
office is going to be very surprised with all the international mail soon.

----

That said, I did write 184 lines of code. (Actually rather a lot, but it
was mostly pure functional code, so easy to write.) That
pops up your text editor on a file with the the trust and group
configurations of repositories, that's stored in the git-annex branch.
Handy for both viewing that stuff all in one place, and changing it.

The real reason for doing that is to provide a nice interface for editing
transfer control expressions, which I'll be adding next.


code:
Started implementing [[transfer_control]]. Although I'm currently calling
the configuration for it "preferred content expressions". (What a mouthful!)

I was mostly able to reuse the Limit code (used to handle parameters like
--not --in otherrepo), so it can already build Matchers for preferred content
expressions in my little Domain Specific Language.

Preferred content expressions can be edited with `git annex vicfg`, which
checks that they parse properly.

The plan is that the first place to use them is not going to be inside the
assistant, but in commands that use the `--auto` parameter, which will use
them as an additional constraint, in addition to the numcopies setting
already used. Once I get it working there, I'll add it to the assistant.

Let's say a repo has a preferred content setting of
"(not copies=trusted:2) and (not in=usbdrive)"

* `git annex get --auto` will get files that have less than 2 trusted
  copies, and are not in the usb drive.
* `git annex drop --auto` will drop files that have 2 or more trusted
  copies, and are not in the usb drive (assuming numcopies allows dropping
  them of course).
* `git annex copy --auto --to thatrepo` run from another repo
  will only copy files that have less than 2 trusted copies. (And if that
  was run on the usb drive, it'd never copy anything!)

There is a complication here.. What if the repo with that preferred content
setting is itself trusted? Then when it gets a file, its number of
trusted copies increases, which will make it be dropped again. :-/

This is a nuance that the numcopies code already deals with, but it's
much harder to deal with it in these complicated expressions. I need to think
about this; the three ideas I'm working on are:

1. Leave it to whoever/whatever writes these expressions to write ones
   that avoid such problems. Which is ok if I'm the only one writing
   pre-canned ones, in practice..
2. Transform expressions into ones that avoid such problems. (For example,
   replace "not copies=trusted:2" with "not (copies=trusted:2 or (in=here and
   trusted=here and copies=trusted:3))"
3. Have some of the commands (mostly drop I think) pretend the drop
   has already happened, and check if it'd then want to get the file back
   again.


code:
[[!comment format=mdwn
 username="http://meep.pl/"
 ip="193.23.174.18"
 subject="Ah, the barber paradox"
 date="2012-10-05T06:51:11Z"
 content="""
Nice. Would (not in=here) be the simplest paradoxical expression?

Is just disregarding the target repo completely during checks a possibility? This would interpret (not copies=trusted:X) as \"not in X *other* trusted repositories, no matter whether we are trusted or not\", and (not in=here) just as \"true\". I think this should generally arrive at the same results as the option 2., but by definition of the expression meaing, not by rewriting.

Alternative 3 (or is my wording different enough to be 3a?) - check that the invariant \"we have all the known files matching our PCE and only these files\" would hold after an operation before actually performing it - could be bistable if done both for gets and drops:

* (not in=here) and we do not have the file -> get thinks \"if we get it, we have a file not matching the PCE\" -> get does not get it;
* (not in=here) and we do have the file -> drop thinks \"if we drop it, there exists a file matching the PCE which we miss\" -> drop does not drop it.

This is not necessarily bad. Checking just for drops should be monostable, I guess, but doesn't it look a bit arbitrary? (Though it would be again equivalent to option 2, wouldn't it? So maybe not that arbitrary.)
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.149"
 subject="comment 2"
 date="2012-10-05T15:08:26Z"
 content="""
Yes, I think checking the future only for drops is both stable and equivilant to the other choices.

Disregarding the target solves the problem for the current set of expressions. There may be future expressions or operations where that does not hold. For example, if move supported --auto (which it does not), you'd need to disregard both sides.

That method would make it impossible to do some possibly useful things. \"in=here or (not copies=3)\"

The real problem with it is that existing options like --copies and --in already take all repos into account, so this would potentally lead to two divergent DSLs being used by git-annex, which would probably be confusing.
"""]]


code:
Fixed the assistant to wait on all the zombie processes that would sometimes
pile up. I didn't realize this was as bad as it was.

Zombies and git-annex have been a problem since I started developing it,
because back then I made some rather poor choices, due to barely knowing
how to write Haskell. So parts of the code that stream input from git commands
don't clean up after them properly. Not normally a problem, because
git-annex reaps the zombies after each file it processes. But this reaping
is not thread-safe; it cannot be used in the assistant.

If I were starting git-annex today, I'd use one of the new Haskell things like
Conduits, that allow for very clean control over finalization of resources.
But switching it to Conduits now would probably take weeks of work; I've not
yet felt it was worthwhile. (Also it's not clear Conduits are the last,
best thing.)

For now, it keeps track of the pids it needs to wait on, and all the code
run by the assistant is zombie-free. However, some code for fsck and unused
that I anticipate the assistant using eventually still has some lurking
zombies.

----

Solved the issue with preferred content expressions and dropping that
I mentioned yesterday. My solution was to add a parameter to specify a set
of repositories where content should be assumed not to be present. When
deciding whether to drop, it can put the current repository in, and then
if the expression fails to match, the content can be dropped.

Using yesterday's example "(not copies=trusted:2) and (not in=usbdrive)",
when the local repo is one of the 2 trusted copies, the drop check will
see only 1 trusted copy, so the expression matches, and so the content will
not be dropped.

I've not tested my solution, but it type checks. :P I'll wire it up to
`get/drop/move --auto` tomorrow and see how it performs.

----

Would preferred content expressions be more readble if they were inverted
(becoming content filtering expressions)?

1. "(not copies=trusted:2) and (not in=usbdrive)" becomes
   "copies=trusted:2 or in=usbdrive"
2. "smallerthan=10mb and include=*.mp3 and exclude=junk/*" becomes
   "largerthan=10mb or exclude=*.mp3" or include=junk/*"
3. "(not group=archival) and (not copies=archival:1)" becomes
   "group=archival or copies=archival:1"

1 and 3 are improved, but 2, less so. It's a trifle weird for "include"
to mean "include in excluded content".

The other reason not to do this is that currently the expressions
can be fed into `git annex find` on the command line, and it'll come
back with the files that would be kept.

Perhaps a middle groud is to make "dontwant" be an alias for "not".
Then we can write "dontwant (copies=trusted:2 or in=usbdrive)"

----

A user told me this:

> I can confirm that the assistant does what it is supposed to do really well. I
> just hooked up my notebook to the network and it starts syncing from notebook to
> fileserver and the assistant on the fileserver also immediately starts syncing
> to the [..] backup

That makes me happy, it's the first quite so real-world success report I've
heard.


code:
[[!comment format=mdwn
 username="http://meep.pl/"
 ip="193.23.174.18"
 subject="Include/exclude"
 date="2012-10-06T08:40:23Z"
 content="""
It seems that example 2 reads weird for filtering because exclude/include say *what to do* while copies/in say *what to test*. An alias (maybe \"glob\") for include  should be acceptable both ways (largerthan=10mb or glob=junk/\* or not glob=\*.mp3).

That said, I would vote for keeping PCEs. And actually the simple \"not (copies=trusted:2 or in=usbdrive)\" does not read significantly worse than with \"dontwant\". (In *my bikeshed* we would have \"all\" == \"glob=\*\" and \"except\" == \"and not\").
"""]]


code:
	git merge watch_

My cursor has been mentally poised here all day, but I've been reluctant to
merge watch into master. It seems solid, but is it correct? I was able to
think up a lot of races it'd be subject to, and deal with them, but did I
find them all?

Perhaps I need to do some automated fuzz testing to reassure myself.
I looked into using [genbackupdata](http://liw.fi/genbackupdata/) to that
end. It's not quite what I need, but could be
[moved in that direction](http://bugs.debian.org/677542). Or I could write
my own fuzz tester, but it seems better to use someone else's, because
a) laziness and b) they're less likely to have the same blind spots I do.

My reluctance to merge isn't helped by the known bugs with files that are
either already open before `git annex watch` starts, or are opened by two
processes at once, and confuse it into annexing the still-open file when one
process closes it.

I've been thinking about just running `lsof` on every file as it's being
annexed to check for that, but in the end, `lsof` is too slow. Since its
check involves trawling through all of /proc, it takes it a good half a
second to check a file, and adding 25 seconds to the time it takes to
process 100 files is just not acceptable.

But an option that could work is to run `lsof` after a bunch of new files
have been annexed. It can check a lot of files nearly as fast as a single
one. In the rare case that an annexed file is indeed still open, it could
be moved back out of the annex. Then when its remaining writer finally
closes it, another inotify event would re-annex it.


code:
[[!comment format=mdwn
 username="http://wiggy.net/"
 nickname="Wichert"
 subject="os compatibility"
 date="2012-06-15T07:19:23Z"
 content="""
A downside of relying on lsof is that you might be painting yourself into a linux corner: other operating systems might not have a lsof or alternative you can rely on. Especially for Windows this might be a worry.
"""]]


code:
[[!comment format=mdwn
 username="http://dieter-be.myopenid.com/"
 nickname="dieter"
 subject="filesystem number of open file handles on a file"
 date="2012-06-15T08:21:37Z"
 content="""
wasn't there some filesystem functionality that could tell you the amount of open file handles on a certain file?  I thought this was tracked per-file too.
Or maybe i'm just confusing it with the number of hard links (which stat can tell you), anyway something to look into.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 3"
 date="2012-06-15T08:58:17Z"
 content="""
I would also be reluctant to use lsof for the sake of non-linux systems or systems that don't have lsof. I've only been playing around with the watch branch of my \"other\" laptop under archlinux. It looks usable, however I would prefer support for OSX before the watch branch gets merged to master ;)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 4"
 date="2012-06-15T10:21:17Z"
 content="""
Corner case, but if the other program finishes writing while you are annexing and your check shows no open files, you are left with bad checksum on a correct file. This \"broken\" file with propagate and the next round of fsck will show that all copies are \"bad\".

Without verifying if this is viable, could you set the file RO and thus block future writes before starting to annex?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.6.135"
 subject="comment 5"
 date="2012-06-15T15:14:52Z"
 content="""
@wichert All this inotify stuff is entirely linux specific AFAIK anyway, so it's find for workarounds to limitations in inotify functionality to also be linux specific.

@dieter I think you're thinking of hard links, filesystems don't track number of open file handles afaik.

@Jimmy, I'm planning to get watch going on freebsd (and hopefully that will also cover OSX), after merging it :)

@Richard, the file is set RO while it's being annexed, so any lsof would come after that point.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.6.135"
 subject="comment 6"
 date="2012-06-15T15:23:21Z"
 content="""
But Rich is right, and I was thinking the same thing earlier this morning, that delaying the lsof allows the writer to change the file and exit, and only fsck can detect the problem then. Setting file permissions doesn't help once a process already has it open for write. Which has put me off the delayed lsof idea unfortunately. lsof *could* be run safely during the intial annexing.
"""]]


code:
To avoid leaking even the size of your encrypted files to cloud storage
providers, add a mode that stores fixed size chunks.

May be a useful starting point for [[deltas]].

May also allow for downloading different chunks of a file concurrently from
multiple remotes.

Also, can allow resuming of interrupted uploads and downloads.

# legacy chunking

Supported by only the webdav and directory special remotes.

Filenames are used for the chunks that make it easy to see which chunks
belong together, even when encryption is used. There is also a chunkcount
file, that similarly leaks information.

It is not possible to enable chunking on a non-chunked remote.

Problem: Two uploads of the same key from repos with different chunk sizes
could lead to data loss. For example, suppose A is 10 mb chunksize, and B
is 20 mb, and the upload speed is the same. If B starts first, when A will
overwrite the file it is uploading for the 1st chunk. Then A uploads the
second chunk, and once A is done, B finishes the 1st chunk and uploads its
second. We now have [chunk 1(from A), chunk 2(from B)].

# new requirements

Every special remote should support chunking. (It does not make sense
to support it for git remotes, but gcrypt remotes should support it.)

S3 remotes should chunk by default, because the current S3 backend fails
for files past a certian size. See [[bugs/Truncated_file_transferred_via_S3]].

The size of chunks, as well as whether any chunking is done, should be
configurable on the fly without invalidating data already stored in the
remote. This seems important for usability (eg, so users can turn chunking
on in the webapp when configuring an existing remote).

Two concurrent uploaders of the same object to a remote should be safe,
even if they're using different chunk sizes.

The legacy chunk method needs to be supported for back-compat, so
keep the chunksize= setting to enable that mode, and add a new chunk=
setting for the new mode.

# obscuring file sizes

To hide from a remote any information about the sizes of files could be
another goal of chunking. At least two things are needed for this:

1. The filenames used on the remote don't indicate which chunks belong
   together.

2. The final short chunk needs to be padded with random data,
   so that a remote sees only encrypted files with uniform sizes
   and cannot make guesses about the kinds of data being stored.

Note that padding cannot completely hide all information from an attacker
who is logging puts or gets. An attacker could, for example, look at the
times of puts, and guess at when git-annex has moved on to
encrypting/decrypting the next object, and so guess at the approximate
sizes of objects. (Concurrent uploads/downloads or random delays could be
added to prevent these kinds of attacks.)

And, obviously, if someone stores 10 tb of data in a remote, they probably
have around 10 tb of files, so it's probably not a collection of recipes..

Given its inneficiencies and lack of fully obscuring file sizes,
padding may not be worth adding, but is considered in the designs below.

# design 1

Add an optional chunk field to Key. It is only present for chunk
2 and above. Ie, SHA256-s12345--xxxxxxx is the first chunk (or whole
object), while SHA256-s12345-c2--xxxxxxx is the second chunk.

On an encrypted remote, Keys are generated with the chunk field, and then
HMAC enrypted.

Note that only using it for chunks 2+ means that git-annex can start by
requesting the regular key, so an observer sees the same request whether
chunked or not, and does not see eg, a pattern of failed requests for
a non-chunked key, followed by successful requests for the chunked keys.
(Both more efficient and perhaps more secure.)

Problem: This makes putting chunks easy. But there is a problem when getting
an object that has been chunked. If the key size is not known, we
cannot tell when we've gotten the last chunk. (Also, we cannot strip
padding.) Note that `addurl` sometimes generates keys w/o size info
(particularly, it does so by design when using quvi).

Problem: Also, this makes `hasKey` hard to implement: How can it know if
all the chunks are present, if the key size is not known?

Problem: Also, this makes it difficult to download encrypted keys, because
we only know the decrypted size, not the encrypted size, so we can't
be sure how many chunks to get, and all chunks need to be downloaded before
we can decrypt any of them. (Assuming we encrypt first; chunking first
avoids this problem.)

Problem: Does not solve concurrent uploads with different chunk sizes.

# design 2

When chunking is enabled, always put a chunk number in the Key,
along with the chunk size.
So, SHA256-1048576-c1--xxxxxxx for the first chunk of 1 megabyte.

Before any chunks are stored, write a chunkcount file, eg
SHA256-s12345-c0--xxxxxxx. Note that this key is the same as the original
object's key, except with chunk number set to 0. This file contains both
the number of chunks, and also the chunk size used. `hasKey` downloads this
file, and then verifies that each chunk is present, looking for keys with
the expected chunk numbers and chunk size.

This avoids problems with multiple writers using different chunk sizes,
since they will be uploading to different files.

Problem: In such a situation, some duplicate data might be stored, not
referenced by the last chunkcount file to be written. It would not be
dropped when the key was removed from the remote.

Note: This design lets an attacker with logs tell the (appoximate) size of
objects, by finding the small files that contain a chunk count, and
correlating when that is written/read and when other files are
written/read. That could be solved by padding the chunkcount key up to the
size of the rest of the keys, but that's very innefficient; `hasKey` is not
designed to need to download large files.

# design 3

Like design 1, but add an encrypted chunk count prefix to the first object.
This needs to be done in a way that does not let an attacker tell if the
object has an encrypted chunk count prefix or not.

This seems difficult; attacker could probably tell where the first encrypted
part stops and the next encrypted part starts by looking for gpg headers,
and so tell which files are the first chunks.

Also, `hasKey` would need to download some or all of the first file.
If all, that's a lot more expensive. If only some is downloaded, an
attacker can guess that the file that was partially downloaded is the
first chunk in a series, and wait for a time when it's fully downloaded to
determine which are the other chunks.

Problem: Two uploads of the same key from repos with different chunk sizes
could lead to data loss. (Same as in design 2.)

# design 4

Use key SHA256-s12345-S1048576-C1--xxxxxxx for the first chunk of 1 megabyte.

Note that keeping the 's'ize field unchanged is necessary because it
disambiguates eg, WORM keys. So a 'S'ize field is used to hold the chunk
size.

Instead of storing the chunk count in the special remote, store it in
the git-annex branch.

The location log does not record locations of individual chunk keys
(too space-inneficient). Instead, look at a chunk log in the
git-annex branch to get the chunk count and size for a key.

`hasKey` would check if any of the logged sets of chunks is
present on the remote. It would also check if the non-chunked key is
present, as a fallback.

When dropping a key from the remote, drop all logged chunk sizes.
(Also drop any non-chunked key.)

As long as the location log and the chunk log are committed atomically,
this guarantees that no orphaned chunks end up on a remote
(except any that might be left by interrupted uploads).

This has the best security of the designs so far, because the special
remote doesn't know anything about chunk sizes. It uses a little more
data in the git-annex branch, although with care (using the same timestamp
as the location log), it can compress pretty well.

## chunk log

Stored in the git-annex branch, this provides a mapping `Key -> [[Key]]`.

Note that a given remote uuid might have multiple sets of chunks (with
different sizes) logged, if a key was stored on it twice using different
chunk sizes. Also note that even when the log indicates a key is chunked,
the object may be stored non-chunked on the remote too.

For fixed size chunks, there's no need to store the list of chunk keys,
instead the log only records the number of chunks (needed because the size
of the parent Key may not be known), and the chunk size.

Example:

	1287290776.765152s e605dca6-446a-11e0-8b2a-002170d25c55:10240 9

Later, might want to support other kinds of chunks, for example ones made
using a rsync-style rolling checksum. It would probably not make sense to
store the full [Key] list for such chunks in the log. Instead, it might be
stored in a file on the remote.

To support such future developments, when updating the chunk log,
git-annex should preserve unparsable values (the part after the colon).

## chunk then encrypt

Rather than encrypting the whole object 1st and then chunking, chunk and
then encrypt.

Reasons:

1. If 2 repos are uploading the same key to a remote concurrently,
   this allows some chunks to come from one and some from another,
   and be reassembled without problems.

2. Also allows chunks of the same object to be downloaded from different
   remotes, perhaps concurrently, and again be reassembled without
   problems.

3. Prevents an attacker from re-assembling the chunked file using details
   of the gpg output. Which would expose approximate
   file size even if padding is being used to obscure it.

Note that this means that the chunks won't exactly match the configured
chunk size. gpg does compression, which might make them a
lot smaller. Or gpg overhead could make them slightly larger. So `hasKey`
cannot check exact file sizes.

If padding is enabled, gpg compression should be disabled, to not leak
clues about how well the files compress and so what kind of file it is.

## chunk key hashing

A chunk key should hash into the same directory structure as its parent
key. This will avoid lots of extra hash directories when using chunking
with non-encrypted keys.

Won't happen when the key is encrypted, but that is good; hashing to the
same bucket then would allow statistical correlation.

## resuming interupted transfers

Resuming interrupted downloads, and uploads are both possible.

Downloads: If the tmp file for a key exists, round it to the chunk size,
and skip forward to the next needed chunk. Easy.

Uploads: Check if the 1st chunk is present. If so, check the second chunk,
etc. Once the first missing chunk is found, start uploading from there.

That adds one extra hasKey call per upload. Probably a win in most cases.
Can be improved by making special remotes open a persistent
connection that is used for transferring all chunks, as well as for
checking hasKey.

Note that this is safe to do only as long as the Key being transferred
cannot possibly have 2 different contents in different repos. Notably not
necessarily the case for the URL keys generated for quvi.

Both **done**.

## parallel

If 2 remotes both support chunking, uploading could upload different chunks
to them in parallel. However, the chunk log does not currently allow
representing the state where some chunks are on one remote and others on
another remote.

Parallel downloading of chunks from different remotes is a bit more doable.


code:
The [[syncing]] design assumes the network is connected. But it's often
not in these pre-IPV6 days, so the cloud needs to be used to bridge between
LANS.

## The cloud notification problem (**done**)

Alice and Bob have repos, and there is a cloud remote they both share.
Alice adds a file; the assistant transfers it to the cloud remote.
How does Bob find out about it?

There are two parts to this problem. Bob needs to find out that there's
been a change to Alice's git repo. Then he needs to pull from Alice's git repo,
or some other repo in the cloud she pushed to. Once both steps are done,
the assistant will transfer the file from the cloud to Bob.

* dvcs-autosync uses xmppp; all repos need to have the same xmpp account
  configured, and send self-messages. An alternative would be to have
  different accounts that join a channel or message each other. Still needs
  account configuration.
* irc could be used. With a default irc network, and an agreed-upon channel,
  no configuration should be needed. IRC might be harder to get through
  some firewalls, and is prone to netsplits, etc. IRC networks have reasons
  to be wary of bots using them. Only basic notifications could be done over
  irc, as it has little security.
* When there's a ssh server involved, code could be run on it to notify
  logged-in clients. But this is not a general solution to this problem.
* pubsubhubbub does not seem like an option; its hubs want to pull down
  a feed over http.

See [[xmpp]] for design of git-annex's use of xmpp for push notifications.

## storing git repos in the cloud **done for XMPP**

Of course, one option is to just use github etc to store the git repo.

Two things can store git repos in Amazon S3:
* <http://gabrito.com/post/storing-git-repositories-in-amazon-s3-for-high-availability>
* <http://wiki.cs.pdx.edu/oss2009/index/projects/gits3.html>

Another option is to not store the git repo in the cloud, but push/pull
peer-to-peer. When peers cannot directly talk to one-another, this could be
bounced through something like XMPP. This is **done** for [[xmpp]]!

Another option: Use <https://github.com/blake2-ppc/git-remote-gcrypt> to store
git repo encrypted on cloud storage.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkq0-zRhubO6kR9f85-5kALszIzxIokTUw"
 nickname="James"
 subject="Cloud Service Limitations"
 date="2012-06-11T02:15:04Z"
 content="""
Hey Joey!

I'm not very tech savvy, but here is my question.
I think for all cloud service providers, there is an upload limitation on how big one file may be.
For example, I can't upload a file bigger than 100 MB on box.net.
Does this affect git-annex at all? Will git-annex automatically split the file depending on the cloud provider or will I have to create small RAR archives of one large file to upload them?

Thanks!
James
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.8.126"
 subject="re: cloud"
 date="2012-06-11T04:48:08Z"
 content="""
Yes, git-annex has to split files for certian providers. I already added support for this as part of my first pass at supporting box.com, see [[tips/using_box.com_as_a_special_remote]].
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn7Oyqusvn0oONFtVhCx5gRAcvPjyRMcBI"
 nickname="Michaël"
 subject="is ftp an option?"
 date="2012-05-30T10:44:12Z"
 content="""
for people only having ftp-access to there storage.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawk9XEh8pxrJxZxIkyK7lWaA7QG1UWt9lgU"
 nickname="Gugelplus"
 subject="OwnCloud"
 date="2012-08-27T20:43:19Z"
 content="""
“Google drive (attractive because it's free, only 5 gb tho)”

Just in case somebody also wants their 5GB of disk space in the cloud, consider using some of the [owncloud providers](http://owncloud.org/providers/). They also offer that amount, but they use free software for everything, using standard protocols (WebDav mostly, because is well supported in all OS).

Git Annex works with them through davfs2, but it would be great if it could support this other program/protocol (OwnCloud/WebDAV) in a more integrated way.
"""]]


code:
[[!comment format=mdwn
 username="http://www.klomp.eu/"
 ip="95.91.241.82"
 subject="Watch also possible with git?"
 date="2012-06-15T17:25:30Z"
 content="""
Hi,

it seems that you put a lot of efforts in handling race conditions. Thats great. I wonder if the watch can also be used with git (i.e. changes are commited into git and not as annex)? I know that other projects follow this idea but why using different tools if the git-annex assistant could handle both...
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawneJXwhacIb0YvvdYFxhlNVpz6Wpg6V7AA"
 nickname="Shayne"
 subject="comment 11"
 date="2012-08-13T00:37:35Z"
 content="""
Yeah definately go with homebrew rather than macports if possible. macports and fink, whilst great systems, have a tendency to sort of create their own alternative-dimension of files within the system that just dont always feel particularly well integrated. As a result \"brew\" has become increasingly more popular to the point its almost ubuquitous now.

Plus its brew-doctor thing is awesome.

The best approach though thats agnostic to distro systems is to simply go for a generic installer.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlu-fdXIt_RF9ggvg4zP0yBbtjWQwHAMS4"
 nickname="Jörn"
 subject="Multiple annexes?"
 date="2012-09-20T16:10:29Z"
 content="""
Thank you for this great piece of software which is now becoming even better with the assistant.

Just one question: will one instance of the assistant be able to track multiple git annex repositories each building up their own network of annexes OR would I need to run multiple instances of the assistant?

Thanks,
Jörn
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 subject="comment 13"
 date="2012-09-20T16:21:11Z"
 content="""
@Jörn, two days ago I added support in the webapp for multiple independent repositories. So you can create independent repos using it, and switch between them from a menu. Under the hood there are multiple git-annex assistant daemons running, but this is an implementation detail; it's not like these use any more memory or other resources than would a single daemon that managed multiple repositories.
"""]]


code:
[[!comment format=mdwn
 username="https://id.koumbit.net/anarcat"
 subject="you rock! & roadmap update?"
 date="2012-09-21T04:25:58Z"
 content="""
joey, you rock. I just want to push on that point - it doesn't seem like there's that many tools that do what git-annex is trying to do out there, and you seem to be doing an incredible job at doing it, so this is great, keep going!

i was wondering - i am glad to see the progress, but it is unclear to me where you actually are in the roadmap. are things going according to plan? are we at month 3? 4? or 1-4? :) just little updates on that roadmap section above would be quite useful!

thanks again!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 subject="comment 15"
 date="2012-09-21T05:25:53Z"
 content="""
We're on month 4 of work, and most of months 1-3 is done well enough for a first pass. Very little of what's listed in month 4 has happened yet, due to my being maybe 2 weeks behind schedule, but bits of [[cloud]] are being planned.

I've made a small adjustment, I think it'll make sense to spend a month on user-driven features before getting into Android.
"""]]


code:
[[!comment format=mdwn
 username="https://launchpad.net/~gdr-go2"
 nickname="gdr-go2"
 subject="Maybe a DEB?2"
 date="2012-09-27T09:44:14Z"
 content="""
Month 3 was all about easy setup, so I kind of expected to download a deb package and just install it, not to download a whole bunch of haskell libraries. Is there a chance that you will release some packages?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.14.141"
 subject="comment 17"
 date="2012-09-27T18:44:11Z"
 content="""
@gdr A package with the assistant is available in Debian unstable.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm7AuSfii_tCkLyspL6Mr0ATlO6OxLNYOo"
 nickname="Georg"
 subject="for OSX, package managers (homebrew and macports) are really second-class"
 date="2013-07-24T07:24:49Z"
 content="""
at least with regards to wide spread use. I agree with Wichert, the average OSX user won't use them. I myself would go with homebrew (because it is far less of a pain than macports), but actually static built binaries that are just dragged from a DMG to your programs folder is the way to go if you can't go osx app store. It's just how most Apple users are wired. PKG would be fine, too, since most will know them, but those are really only needed if you need to place stuff in /Library/Frameworks or things like that. If it can be done as a all-inclusive OSX .app, do it that way and just pack it up in a .DMG.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlnoH5btjn_BLib3_IhES5uMhrzuOiwCYo"
 nickname="András"
 subject="windows port"
 date="2013-07-25T07:50:19Z"
 content="""
I'd love to use the Windows port of the assistant by Christmas. Keep up the good work! :)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 1"
 date="2012-06-02T12:06:37Z"
 content="""
Will statically linked binaries be provided for say Linux, OSX and *BSD?  I think having some statically linked binaries will certainly help and appeal to a lot of users.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="2001:4978:f:21a::2"
 subject="comment 20"
 date="2013-07-25T18:22:51Z"
 content="""
@Georg, DMG have been built for OSX for a long time. See [[install/OSX]]
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 subject="comment 2"
 date="2012-06-04T19:45:00Z"
 content="""
Jimmy, I hope to make it as easy as possible to install. I've been focusing on getting it directly into popular Linux distributions, rather than shipping my own binary. The OSX binary is static, and while I lack a OSX machine, I would like to get it easier to distribute to OSX users.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 3"
 date="2012-06-07T20:22:55Z"
 content="""
I'd agree getting it into the main distros is the way to go, if you need OSX binaries, I could volunteer to setup an autobuilder to generate binaries for OSX users, however it would rely on users to have macports with the correct ports installed to use it (things like coreutils etc...)

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 subject="comment 4"
 date="2012-06-08T01:56:52Z"
 content="""
I always appreciate your OSX work Jimmy...

Could it be put into macports?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 5"
 date="2012-06-08T07:22:34Z"
 content="""
In relation to macports, I often found that haskell in macports are often behind other distros, and I'm not willing to put much effort into maintaining or updating those ports. I found that to build git-annex, installing macports manually and then installing haskell-platform from the upstream to be the best way to get the most up to date dependancies for git-annex.

fyi in macports ghc is at version 6.10.4 and haskell platform is at version 2009.2, so there are a significant number of ports to update.

I was thinking about this a bit more and I reckon it might be easier to try and build a self contained .pkg package and have all the needed binaries in a .app styled package, that would work well when the webapp comes along. I will take a look at it in a week or two (currently moving house so I dont have much time)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 6"
 date="2012-06-08T15:21:18Z"
 content="""
It's not much for now... but see <http://www.sgenomics.org/~jtang/gitbuilder-git-annex-x00-x86_64-apple-darwin10.8.0/> I'm ignoring the debian-stable and pristine-tar branches for now, as I am just building and testing on osx 10.7.

Hope the autobuilder will help you develop the OSX side of things without having direct access to an osx machine! I will try and get gitbuilder to spit out appropriately named tarballs of the compiled binaries in a few days when I have more time.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.8.126"
 subject="comment 7"
 date="2012-06-09T18:07:51Z"
 content="""
Thanks, that's already been useful to me. You might as well skip the debian-specific \"bpo\" tags too.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawldKnauegZulM7X6JoHJs7Gd5PnDjcgx-E"
 nickname="Matt"
 subject="Homebrew instead of MacPorts"
 date="2012-06-22T04:26:02Z"
 content="""
[Homebrew] is a much better package manager than MacPorts IMO.

[Homebrew]: http://mxcl.github.com/homebrew/
"""]]


code:
[[!comment format=mdwn
 username="http://wiggy.net/"
 nickname="Wichert"
 subject="macports"
 date="2012-06-12T13:00:34Z"
 content="""
The average OSX user has a) no idea what macports is, and b) will not be able to install it. Anything that requires a user to do anything with a commandline (or really anything other than using a GUI installer) is effectively a dealbreaker. For our use cases OSX is definitely a requirement, but it must only use standard OSX installation methods in order to be usable. Being in the appstore would be ideal, but standard dmg/pkg installers are still common enough that they are also acceptable.

FWIW this is the same reason many git GUIs were not usable for our OSX users: they required separate installation of the git commandline tools.
"""]]


code:
Add to the [[webapp]] some configuration of git-annex.

There are some basic settings that pass through to `git config`, things
like how much disk space to leave free, how many copies to ensure are kept
of files, etc.

The meat of the configuration will be in configuration assistants that walk
through setting up common use cases.

* Create a repository (run when the web app is started without a configured
  repository too). **done**
* Clone this repo to a USB drive or other removable drive. **done**
* Make a bare repo on a remote ssh server **done**
* Clone this repo to another host. (Needs [[pairing]]) **done**
* Set up Amazon S3. **done**
* Set up encrypted rsync remote. **done**
* Rsync.net special case **done**
* Set up gpg encryption key; gpg key distribution.
* I lost my USB drive!
* etc -- many more possibilities


code:
Speed up syncing of modified versions of existing files.

One simple way is to find the key of the old version of a file that's
being transferred, so it can be used as the basis for rsync, or any
other similar transfer protocol.

For remotes that don't use rsync, use a rolling checksum based chunker,
such as BuzHash. This will produce [[chunks]], which can be stored on the
remote as regular Keys -- where unlike the fixed size chunk keys, the
SHA256 part of these keys is the checksum of the chunk they contain.

Once that's done, it's easy to avoid uploading chunks that have been sent
to the remote before.

When retriving a new version of a file, there would need to be a way to get
the list of chunk keys that constitute the new version. Probably best to
store this list on the remote. Then there needs to be a way to find which
of those chunks are available in locally present files, so that the locally
available chunks can be extracted, and combined with the chunks that need
to be downloaded, to reconstitute the file.

To find which chucks are locally available, here are 2 ideas:

1. Use a single basis file, eg an old version of the file. Re-chunk it, and
   use its chunks.  Slow, but simple.
2. Some kind of database of locally available chunks. Would need to be kept
   up-to-date as files are added, and as files are downloaded.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkfHTPsiAcHEEN7Xl7WxiZmYq-vX7azxFY"
 nickname="Vincent"
 subject="zsync?"
 date="2012-09-10T12:35:45Z"
 content="""
zsync.moria.org.uk may have broken some of the ground here.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm72W-CH7bzZ8uHvaw1KJGrToPSQDNBnIA"
 nickname="Beni"
 subject="bup splitting"
 date="2013-08-05T17:13:14Z"
 content="""
bup splits files into chunks based on rolling sums.
https://github.com/apenwarr/bup/blob/master/DESIGN
For small changes to big files, this would improve not only transfer speed but also repository size.
And git-annex already supports bup remotes.  What's missing?
"""]]


code:
While dropbox allows modifying files in the folder, git-annex freezes
them upon creation, using symlinks.

This is a core design simplification of git-annex.
But it is problematic too:

* To allow directly editing files in its folder, something like [[todo/smudge]] is
  needed, to get rid of the symlinks that stand in for the files.
* OSX seems to have a [[lot_of_problems|bugs/OSX_alias_permissions_and_versions_problem]]
  with stupid programs that follow symlinks and present the git-annex
  hash filename to the user.
* FAT sucks and doesn't support symlinks at all, so [[Android]] can't
  have regular repos on it.

One approach for this would be to hide the git repo away somewhere,
and have the git-annex assistant watch a regular directory, with
regular files.

There would have to be a mapping from files to git-annex objects.
And some intelligent way to determine when a file has been changed
and no longer corresponds to its object. (Not expensive hashing every time,
plz.)

Since this leaves every file open to modification, any such repository
probably needs to be considered untrusted by git-annex. So it doesn't
leave its only copy of a file in such a repository, but instead
syncs it to a proper git-annex repository.

The assistant would make git commits still, of symlinks. It can already do
that with without actual symlinks existing on disk. More difficult is
handling merging; git merge wants a real repository with files it can
really operate on. The assistant would need to calculate merges on its own,
and update the regular directory to reflect changes made in the merge.

Another massive problem with this idea is that it doesn't allow for
[[partial_content]]. The symlinks that everyone loves to hate on are what
make it possible for the contents of some files to not be present on
disk, while the files are still in git and can be retreived as desired.
With real files, some other standin for a missing file would be needed.
Perhaps a 0 length, unreadable, unwritable file? On systems that
support symlinks it could be a broken symlink like is used now, that
is converted to a real file when it becomes present.

## concrete design

* Enable with annex.direct
* Use .git/ for the git repo, but `.git/annex/objects` won't be used
  for object storage.
* `git status` and similar will show all files as type changed, and
  `git commit` would be a very bad idea. Just don't support users running
  git commands that affect the repository in this mode. Probably.
* However, `git status` and similar also will show deleted and new files,
  which will be helpful for the assistant to use when starting up.
* Cache the mtime, size etc of files, and use this to detect when they've been
  modified when the assistant was not running. This would only need to be
  checked at startup, probably.
* Use dangling symlinks for standins for missing content, as now.
  This allows just cloning one of these repositories normally, and then
  as the files are synced in, they become real files.
* Maintain a local mapping from keys to files in the tree. This is needed
  when sending/receiving/dropping keys to know what file to access.
  Note that a key can map to multiple files. And that when a file is
  deleted or moved, the mapping needs to be updated.
* May need a reverse mapping, from files in the tree to keys? TBD
  (Currently, getting by looking up symlinks using `git cat-file`)
  (Needed to make things like `git annex drop` that want to map from the
  file back to the key work.)
* The existing watch code detects when a file gets closed, and in this
  mode, it could be a new file, or a modified file, or an unchanged file.
  For a modified file, can compare mtime, size, etc, to see if it needs
  to be re-added.
* The inotify/kqueue interface does not tell us when a file is renamed.
  So a rename has to be treated as a delete and an add, so can have a lot
  of overhead, to re-hash the file.
* Note that this could be used without the assistant, as a git remote
  that content is transferred to and from. Without the assistant, changes
  to files in this remote would not be noticed and committed, unless
  a git-annex command were added to do so.
  Getting it basically working as a remote would be a good 1st step.
* It could also be used without the assistant as a repository that
  the user uses directly. Would need some git-annex commands
  to merge changes into the repo, update caches, and commit changes.
  This could all be done by "git annex sync".

## TODO

* kqueue does not deliver an event when an existing file is modified.
  This doesn't affect OSX, which uses FSEvents now, but it makes direct
  mode assistant not 100% on other BSD's.

## done

* `git annex sync` updates the key to files mappings for files changed,
  but needs much other work to handle direct mode:
  * Generate git commit, without running `git commit`, because it will
    want to stage the full files. **done**
  * Update location logs for any files deleted by a commit. **done**
  * Generate a git merge, without running `git merge` (or possibly running
    it in a scratch repo?), because it will stumble over the direct files.
    **done**
  * Drop contents of files deleted by a merge (including updating the
    location log), or if we cannot drop,
    move their contents to `.git/annex/objects/`.  **no**  .. instead,
    avoid ever losing file contents in a direct mode merge. If the file is
    deleted, its content is moved back to .git/annex/objects, if necessary.
  * When a merge adds a symlink pointing at a key that is present in the
    repo, replace the symlink with the direct file (either moving out
    of `.git/annex/objects/` or hard-linking if the same key is present
    elsewhere in the tree. **done**
  * handle merge conflicts on direct mode files **done**
* support direct mode in the assistant (many little fixes)

* Deal with files changing as they're being transferred from a direct mode
  repository to another git repository. The remote repo currently will
  accept the bad data and update the location log to say it has the key.

  This affects both special remotes and git remotes.

  For special remotes,
  it seems the best that could be done is to have an error unwind action
  passed to `sendAnnex` that is called if the file is modified as it's
  transferred. That would then remove the probably corrupted file from the
  remote. (The full transfer would still run, unless there was also a way
  to cancel an in progress transfer.) **done**

  (With the above, there is some potential for the bad content being
  downloaded from the special remote into another repo. This would only
  happen if the other repo for some reason thinks the special remote
  has the content. Since the location log would not be updated until the
  transfer is successful, this should not happen.)

  For local git remotes, need to check the direct mode file after it's
  copied and before it's put into place as a key's content. **done**
  (untested)

  `git-annex-shell sendkey` needs to do something if it sent bad
  data. This seems to not need protocol changes; it can just detect
  the problem and exit nonzero. Would need to do something to clean up
  the temp file, which is probably corrupt. (Could in future use it as a
  basis for transferring the new key..) **done**

  For git remotes, added a flag to `git-annex-shell recvkey` (using a field
  after the "--" to remain back-compat). With this flag, after receiving
  the data, the remote fscks the data. This is not optimal, but avoids
  needing another round-trip, or a protocol change.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawn-KDr_Z4CMkjS0v_TxQ08SzAB5ecHG3K0"
 nickname="Glen"
 subject="Modifying files in a folder watched by assistant"
 date="2012-09-19T14:18:54Z"
 content="""
I think this feature is really important.  For my use cases, if I can't modify files that are in the folder that is kept in sync, then I would just use rsync or manually keep copies on multiple systems..

I might be missing something, but tried a git annex unlock on the file, but if the assistant is running, it seems to revert this back.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawknpbpx4uDElXRzJ4kPXuJU1KdcslOI6go"
 nickname="Rodrigo"
 subject="I strongly agree with Glen"
 date="2012-10-03T14:07:40Z"
 content="""
And if your goal is to build an app \"like dropbox\" it is a must have feature.
Thanks for your work. You are doing great!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.149"
 subject="comment 3"
 date="2012-10-05T00:43:11Z"
 content="""
@Glen, for what it's worth, you can now manually \"git annex unlock\" a file and it'll stay unlocked until you manually \"git annex add\" the new version.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlJEI45rGczFAnuM7gRSj4C6s9AS9yPZDc"
 nickname="Kevin"
 subject="unlock not behaving as you describe"
 date="2012-11-05T22:35:35Z"
 content="""
@joeyh,

I'm not seeing the behavior you describe re \"git annex unlock\" when using the assistant.  After unlocking and writing the file, git annex assistant commits the changes and recreates the symlink from under me which is annoying.

I'm running OS X 10.7.5 using the bundled git-annex.app version 3.20121017.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.27"
 subject="comment 5"
 date="2012-11-05T23:40:50Z"
 content="""
Well, you're able to unlock and modify the file -- that's what I fixed.

Committing changes to files is what the assistant does..
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm5iosFbL2By7UFeViqkc6v-hoAtqILeDA"
 nickname="Laszlo"
 subject="comment 6"
 date="2012-12-03T05:33:44Z"
 content="""
> One approach for this would be to hide the git repo away somewhere, and have the git-annex assistant watch a regular directory, with regular files.

I think this would be the single most important feature added to git-annex.
1. It could be used to backup regularly a smb share over the network.
2. For those who dont trust a program who modifies their files directly.
(I had a 3GB permanent data loss, when trying to push a 10GB directory into git annex.)
3. Android

I do hope the above method will be implemented for Android too, so its like two birds with one stone.
(Im really for option 2 here. I dont like the very idea of modifying my files directly.
Git does not do it either, I have disk space (if not I buy more disks), rather then risquing losing important datas)

Best,
 Laszlo

"""]]


code:
The assistant should help the user recover their repository when things go
wrong.

[[!toc ]]

## dangling lock files

There are a few ways a git repository can get broken that are easily fixed.
One is left over index.lck files. When a commit to a repository fails,
check that nothing else is using it, fix the problem, and redo the commit.

* **done** for .git/annex/index.lock, can be handled safely and automatically.
* **done** for .git/index.lock, only when the assistant is starting up.
* What about local remotes, eg removable drives? git-annex does attempt
  to commit to the git-annex branch of those. It will use the automatic
  fix if any are dangling. It does not commit to the master branch; indeed
  a removable drive typically has a bare repository.
  However, it does a scan for broken locks anyway if there's a problem
  syncing. **done**
* What about git-annex-shell? If the ssh remote has the assistant running,
  it can take care of it, and if not, it's a server, and perhaps the user
  should be required to fix up if it crashes during a commit. This should
  not affect the assistant anyway.
* **done** Seems that refs can also have stale lock files, for example
  '/storage/emulated/legacy/DCIM/.git/refs/remotes/flick_phonecamera/synced/git-annex.lock'
  All git lock files are now handled (except gc lock files).

## incremental fsck

Add webapp UI to enable incremental fsck **done**

Of course, incremental fsck will run as an niced (and ioniced) background
job. There will need to be a button in the webapp to stop it, in case it's
annoying. **done**

When fsck finds a damanged file, queue a download of the file from a
remote. **done**

Detect when a removable drive is connected in the Cronner, and check
and try to run its remote fsck jobs. **done** (Same mechanism will work for
network remotes becoming connected.)

TODO: If no accessible remote has a file that fsck reported missing,
prompt the user to eg, connect a drive containing it. Or perhaps this is a
special case of a general problem, and the webapp should prompt the user
when any desired file is available on a remote that's not mounted?

## git-annex-shell remote fsck

TODO: git-annex-shell fsck support, which would allow cheap fast fscks
of ssh remotes.

Would be nice; otherwise remote fsck is too expensive (downloads
everything) to have the assistant do.

Note that Remote.Git already tries to use this, but the assistant does not
call it for non-local remotes.

## git fsck and repair

Add git fsck to scheduled self fsck **done**

TODO: git fsck on ssh remotes? Probably not worth the complexity..

TODO: If committing to the repository fails, after resolving any dangling
lock files (see above), it should git fsck. This is difficult, because
git commit will also fail if the commit turns out to be empty, or due to
other transient problems.. So commit failures are currently ignored by the
assistant.

If git fsck finds problems, launch git repository repair. **done**

git annex fsck --fast at end of repository repair to ensure
git-annex branch is accurate. **done**

If syncing with a local repository fails, try to repair it. **done**

TODO: "Repair" gcrypt remotes, by removing all refs and objects,
and re-pushing. (Since the objects are encrypted data, there is no way
to pull missing ones from anywhere..)
Need to preserve gcrypt-id while doing this!

TODO: along with displaying alert when there is a problem detected
by consistency check, send an email alert. (Using system MTA?)

## nudge user to schedule fscks

Make the webapp encourage users to schedule fscks of their
local repositories. The goal here was that it should not be obnoxious about
repeatedly pestering the user to set that up, but should still encourage
anyone who cares to set it up.

Maybe: Display a message only once per week, and only after the repository
has existed for at least one full day. But, this will require storing
quite a lot of state.

Or: Display a message whenever a removable drive is detected to have been
connected. I like this, but what about nudging the main repo? Could do it
every webapp startup, perhaps? **done**

There should be a "No thanks" button that prevents it nudging again for a
repo. **done**

## git repository repair

There are several ways git repositories can get damanged.

The most common is empty files in .git/annex/objects and commits that refer
to those objects. When the objects have not yet been pushed anywhere.
I've several times recovered from this manually by
removing the bad files and resetting to before the commits that referred to
them. Then re-staging any divergence in the working tree. This could
perhaps be automated.

As long as the git repository has at least one remote, another method is to
clone the remote, sync from all other remotes, move over .git/config and
.git/annex/objects, and tar up the old broken git repo and `git annex add`
it. This should be automatable and get the user back on their feet. User
could just click a button and have this be done.

This is useful outside git-annex as well, so make it a
git-recover-repository command.

### detailed design

Run `git fsck` and parse output to find bad objects. **done** Note that
fsck may fall over and fail to print out all bad objects, when
files are corrupt. So if the fsck exits nonzero, need to collect all
bad objects it did find, and:

1. If the local repository contains packs, the packs may be corrupt.
   So, start by using `git unpack-objects` to unpack all
   packs it can handle (which may include parts of corrupt packs)
   back to loose objects. And delete all packs. **done**
2. Delete all loose corrupt objects. **done**

Repeat until fsck finds no new problems. **done**

Check if there's a remote. If so, and if the bad objects are all
present on it, can simply get all bad objects from the remote,
and inject them back into .git/objects to recover:

3. Make a new (bare) clone from the remote.
   (Note: git does not seem to provide a way to fetch specific missing
   objects from the remote. Also, cannot use `--reference` against
   a repository with missing refs. So this seems unavoidably
   network-expensive.) **done**
5. Rsync objects over. (Turned out to work better than git-cat-file,
   because we don't have to walk the graph to add missing objects.)
   **done**
6. If each bad object was able to be repaired this way, we're done!
   (If not, can reuse the clone for getting objects from the next remote.)
   **done**

If some missing objects cannot be recovered from remotes, find commits in each
local branch that are broken by all remaining missing objects. Some of this can
be parsed from git fsck output, but for eg blobs, the commits need to
be walked to walk the trees, to find trees that refer to the blobs. **done**

For each branch that is affected, look in the reflog and/or `git log
$branch` to find the last good commit that predates all broken commits. (If
the head commit of a branch is broken, git log is not going to show
anything useful, but the reflog can be used to find past refs for the
branch -- have to first delete the .git/HEAD file if it points to the
broken ref.) **done**

The basic idea then is to reset the branch to the last good commit
that was found for it.

* For the HEAD branch, can just reset it. (If no last good commit was found
  for the HEAD branch, reset it to a dummy empty commit.) This will
  leave git showing any changes made since then as staged in the index and
  uncommitted. Or if the index is missing/corrupt, any files in the tree will
  show as modified and uncommitted. User (or git-annex assistant) can then
  commit as appropriate. Print appropriate warning message. **done**
* Special handling for git-annex branch and index. **done**
* Remote tracking branches can just be removed, and then `git fetch`
  from the remote, which will re-download missing objects from it and
  reinstate the tracking branch. **done**
* For other branches, reset them to last good commit, or delete
  if none was found. **done**
* (Decided not to touch tags.)

The index file can still refer to objects that were missing.
Rewrite to remove them. **done**


code:
[[!comment format=mdwn
 username="http://nullroute.eu.org/~grawity/"
 nickname="Mantas"
 subject="comment 1"
 date="2013-10-19T15:50:47Z"
 content="""
The restriction on fetching over the Git protocol is, partly, for security reasons – e.g. if one accidentally pushes a commit with private data, and then `push --force`'s a cleaned-up version, Git needs to prevent anyone from downloading the old commit by just giving its SHA1 (e.g. obtained from an IRC/email push notification). So it restricts fetching to the tips of any ref. (I've been told that it could check if the given object is merely *reachable* from any ref, but it doesn't do so for performance reasons.)

git 1.8 has a minor way to relax this requirement – it allows giving a SHA1 to `git fetch` (although I think the protocol already worked this way), and it allows refs to be hidden server-side but still remain fetchable, so in theory there could be a (hidden) ref for every object, for easy fetching...
"""]]


code:
Encrypted git remotes are now possible
using [git-remote-gcrypt](https://github.com/joeyh/git-remote-gcrypt).

There are at least two use cases for this in the assistant:

* Storing an encrypted git repository on a local drive. **done**
* Or on a remote server. This could even allow using github. But more
  likely would be a shell server that has git-annex-shell on it so can
  also store file contents, and which is not trusted with unencrypted data.
  **done**

git-remote-gcrypt is already usable with git-annex. What's needed is
to make sure it's installed (ie, get it packaged into distros or embedded
into git-annex), and make it easy to set up from the webapp. **done**

Hmm, this will need gpg key creation, so would also be a good opportunity
to make the webapp allow using that for special remotes too.

One change is needed in git-annex core.. It currently does not support
storing encrypted files on git remotes, only on special remotes. Perhaps
the way to deal with this is to make it consider git-remote-grypt remotes
to be a special remote type? **done**


code:
Currently the assistant sets up a shared encryption key, which is checked
into git, so anyone who gets the repository can decrypt files that are
stored encrypted on special remotes.

To support using gpg keys in the assistant, we need some things:

1. Help user set up a gpg key if they don't have one. This could be a
   special-purpose key dedicated to being used by git-annex. It might be
   nice to leave the user with a securely set up general purpose key,
   but that would certainly preclude prompting for its password in the
   webapp. Indeed, the password prompt is the main problem here.
   Best solution would be to get gpg agent working on all supported
   platforms.

   Update: For now, git-annex only assists in generating gpg keys that are
   intended to only be used to encrypt a repo.

2. After generating a gpg key, back it up. It might be the only way
   some data is accessible.

   One way I'm considering is generating a QR code
   of the key, which could be printed to paper. Preliminary results
   are good; a 4096 bit secret key fits in a QR code (a secret key
   with many subkeys may not). Debian has command-line utilities that
   can generate and read such a QR code.

   Another way would be to use shamir secret sharing to split the key into
   N peices and send each one to one of the user's repos.

3. Help user learn the gpg keys of people they want to share their repo
   with, and give them access. If the public key was recorded in the git-annex
   branch, this could be easily determined when sharing repositories with
   friends. Or, use MonkeySphere, or Monkeysign..

-----

Another gpg key security thing is that currently git-annex stores
crypto creds in memory while it's running. Should use locked memory. See
<https://github.com/vincenthz/hs-securemem> and
<https://github.com/vincenthz/hs-securemem/issues/1>


code:
[[!comment format=mdwn
 username="gernot"
 ip="85.197.15.125"
 subject="Paperkey"
 date="2013-12-05T11:37:02Z"
 content="""
Regarding backups, have you considered using [paperkey](http://www.jabberwocky.com/software/paperkey/) (also [in Debian](http://packages.debian.org/search?keywords=paperkey))?
It only stores a minimal amount of key data and formats it in a human-readable way.

The result is mainly meant to be printed but `paperkey` would probably also be a good way to keep the size of QR codes down.

I've actually recovered a key from a such a printed backup (using OCR) and it worked great thanks to line-wise checksums.
Maybe you could create paperkey PDF files with a proper OCR font through the web app?

"""]]


code:
"git annex watch" command, which runs, in the background, watching via
inotify for changes, and automatically annexing new files, etc. Now
available!

[[!toc]]

## known bugs

* Kqueue has to open every directory it watches, so too many directories
  will run it out of the max number of open files (typically 1024), and fail.
  I may need to fork off multiple watcher processes to handle this.
  See [[bug|bugs/Issue_on_OSX_with_some_system_limits]]. (Does not affect
  OSX any longer, only other BSDs).

## beyond Linux

I'd also like to support OSX and if possible the BSDs.

* kqueue ([haskell bindings](http://hackage.haskell.org/package/kqueue))
  is supported by FreeBSD, OSX, and other BSDs.

  In kqueue, to watch for changes to a file, you have to have an open file
  descriptor to the file. This wouldn't scale.

  Apparently, a directory can be watched, and events are generated when
  files are added/removed from it. You then have to scan to find which
  files changed. [example](https://developer.apple.com/library/mac/#samplecode/FileNotification/Listings/Main_c.html#//apple_ref/doc/uid/DTS10003143-Main_c-DontLinkElementID_3)

  Gamin does the best it can with just kqueue, supplimented by polling.
  The source file `server/gam_kqueue.c` makes for interesting reading.
  Using gamin to do the heavy lifting is one option.
  ([haskell bindings](http://hackage.haskell.org/package/hlibfam) for FAM;
  gamin shares the API)

  kqueue does not seem to provide a way to tell when a file gets closed,
  only when it's initially created. Poses problems..

  * [man page](http://www.freebsd.org/cgi/man.cgi?query=kqueue&apropos=0&sektion=0&format=html)
  * <https://github.com/gorakhargosh/watchdog/blob/master/src/watchdog/observers/kqueue.py> (good example program)

  *kqueue is now supported*

* hfsevents ([haskell bindings](http://hackage.haskell.org/package/hfsevents))
  is OSX specific.

  Originally it was only directory level, and you were only told a
  directory had changed and not which file. Based on the haskell
  binding's code, from OSX 10.7.0, file level events were added.

  This will be harder for me to develop for, since I don't have access to
  OSX machines..

  hfsevents does not seem to provide a way to tell when a file gets closed,
  only when it's initially created. Poses problems..

  * <https://developer.apple.com/library/mac/#documentation/Darwin/Conceptual/FSEvents_ProgGuide/Introduction/Introduction.html>
  * <http://pypi.python.org/pypi/MacFSEvents/0.2.8> (good example program)
  * <https://github.com/gorakhargosh/watchdog/blob/master/src/watchdog/observers/fsevents.py> (good example program)

  *hfsevents is now supported*

* Windows has a Win32 ReadDirectoryChangesW, and perhaps other things.

  It was easy to get watching to work in windows. But there is no lsof,
  to check if a file can safely be added. So, need to carefully consider
  how to make adding a file safe in windows.

  Without lsof, an InodeCache is generated in "lockdown" (which doesn't
  do anything to prevent new writers), and is compared with the stat of the
  file after it's ingested (and checksummed). This will detect many changes
  to files, which change the size or mtime.

  So, we have 2 cases to worry about.

  1. A process has the file open for write as it's added, does not change
     it until the add is done.

     As long as an event is generated once the file does get closed, this
     is fine -- the modified version will be re-added. And such events are
     indeed generated on windows.

  2. A process has the file open for write as it's added, and changes it
     in some way that does not affect size or mtime.

     If an event is generated when the file does get closed, this is the
     same as a scenario where a process opens the file after it's added,
     makes such a change, and closes it. In either case, a file closed
     event is generated, and the Watcher will not detect any change
     using the inode cache, so will not re-add the file.

     So, this scenario is a potential problem, but it seems at least
     unlikely that a program would modify a file without affecting its
     mtime. Note that this same scenario can happen even with lsof, and
     even on linux (although on linux the InodeCache includes an actual
     inode, which might detect the change too).

  Conclusion: It's probably ok to run without lsof on Windows.

  Corrolary: lsof might not generally be needed in direct mode, on
  systems that do generate file close events (but not when
  eventsCoalesce).
  The same arguments given above seem to apply to !Windows. Note that lsof
  is needed in indirect mode, as discussed below.

  **windows is now supported**

## the races

Many races need to be dealt with by this code. Here are some of them.

* File is added and then removed before the add event starts.

  Not a problem; The add event does nothing since the file is not present.

* File is added and then removed before the add event has finished
  processing it.

  **Minor problem**; When the add's processing of the file (checksum and so
  on) fails due to it going away, there is an ugly error message, but
  things are otherwise ok.

* File is added and then replaced with another file before the annex add
  moves its content into the annex.

  Fixed this problem; Now it hard links the file to a temp directory and
  operates on the hard link, which is also made unwritable.

* File is added and then replaced with another file before the annex add
  makes its symlink.

  **Minor problem**; The annex add will fail creating its symlink since
  the file exists. There is an ugly error message, but the second add
  event will add the new file.

* File is added and then replaced with another file before the annex add
  stages the symlink in git.

  Now fixed; `git annex watch` avoids running `git add` because of this
  race. Instead, it stages symlinks directly into the index, without
  looking at what's currently on disk.

* Link is moved, fixed link is written by fix event, but then that is
  removed by the user and replaced with a file before the event finishes.

  Now fixed; same fix as previous race above.

* File is removed and then re-added before the removal event starts.

  Not a problem; The removal event does nothing since the file exists,
  and the add event replaces it in git with the new one.

* File is removed and then re-added before the removal event finishes.

  Not a problem; The removal event removes the old file from the index, and
  the add event adds the new one.

* Symlink appears, but is then deleted before it can be processed.

  Leads to an ugly message, otherwise no problem:

	./me: readSymbolicLink: does not exist (No such file or directory)

  Here `me` is a file that was in a conflicted merge, which got
  removed as part of the resolution. This is probably coming from the watcher
  thread, which sees the newly added symlink (created by the git merge),
  but finds it deleted (by the conflict resolver) by the time it processes it.

## done

- on startup, add any files that have appeared since last run **done**
- on startup, fix the symlinks for any renamed links **done**
- on startup, stage any files that have been deleted since last run
  (seems to require a `git commit -a` on startup, or at least a
  `git add --update`, which will notice deleted files) **done**
- notice new files, and git annex add **done**
- notice renamed files, auto-fix the symlink, and stage the new file location
  **done**
- handle cases where directories are moved outside the repo, and stop
  watching them **done**
- when a whole directory is deleted or moved, stage removal of its
  contents from the index **done**
- notice deleted files and stage the deletion
  (tricky; there's a race with add since it replaces the file with a symlink..)
  **done**
- Gracefully handle when the default limit of 8192 inotified directories
  is exceeded. This can be tuned by root, so help the user fix it.
  **done**
- periodically auto-commit staged changes (avoid autocommitting when
  lots of changes are coming in) **done**
- coleasce related add/rm events for speed and less disk IO **done**
- don't annex `.gitignore` and `.gitattributes` files **done**
- run as a daemon **done**
- A process has a file open for write, another one closes it,
  and so it's added. Then the first process modifies it.

  Or, a process has a file open for write when `git annex watch` starts
  up, it will be added to the annex. If the process later continues
  writing, it will change content in the annex.

  This changes content in the annex, and fsck will later catch
  the inconsistency.

  Possible fixes:

  * Somehow track or detect if a file is open for write by any processes.
    `lsof` could be used, although it would be a little slow.

    Here's one way to avoid the slowdown: When a file is being added,
    set it read-only, and hard-link it into a quarantine directory,
    remembering both filenames.
    Then use the batch change mode code to detect batch adds and bundle
    them together.
    Just before committing, lsof the quarantine directory. Any files in
    it that are still open for write can just have their write bit turned
    back on and be deleted from quarantine, to be handled when their writer
    closes. Files that pass quarantine get added as usual. This avoids
    repeated lsof calls slowing down adds, but does add a constant factor
    overhead (0.25 seconds lsof call) before any add gets committed. **done**

  * Or, when possible, making a copy on write copy before adding the file
    would avoid this.
  * Or, as a last resort, make an expensive copy of the file and add that.
  * Tracking file opens and closes with inotify could tell if any other
    processes have the file open. But there are problems.. It doesn't
    seem to differentiate between files opened for read and for write.
    And there would still be a race after the last close and before it's
    injected into the annex, where it could be opened for write again.
    Would need to detect that and undo the annex injection or something.

- If a file is checked into git as a normal file and gets modified
  (or merged, etc), it will be converted into an annexed file.
  See [[blog/day_7__bugfixes]]. **done**; we always check ls-files now
- When you `git annex unlock` a file, it will immediately be re-locked.
  See [[bugs/watcher_commits_unlocked_files]]. Seems fixed now?


code:
[[!comment format=mdwn
 username="http://ciffer.net/~svend/"
 subject="comment 1"
 date="2012-06-04T19:42:07Z"
 content="""
I would find it useful if the watch command could 'git add' new files (instead of 'git annex add') for certain repositories.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 subject="comment 2"
 date="2012-06-04T19:46:03Z"
 content="""
I think it's already on the list: \"configurable option to only annex files meeting certian size or filename criteria\" -- files not meeting those criteria would just be git added.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 3"
 date="2012-06-17T08:52:32Z"
 content="""
In relation to OSX support, hfsevents (or supporting hfs is probably a bad idea), its very osx specific and users who are moving usb keys and disks between systems will probably end up using fat32/exfat/vfat disks around. Also if you want I can lower the turn around time for the OSX auto-builder that I have setup to every 1 or 2mins? would that help?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.2.147"
 subject="comment 4"
 date="2012-06-17T16:39:43Z"
 content="""
hfsevents seems usable, git-annex does not need to watch for file changes on remotes on other media.

But, trying kqueue first.

You could perhaps run the autobuilder on a per-commit basis..
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 5"
 date="2012-06-17T21:42:59Z"
 content="""
okay, I've gotten gitbuilder to poll the git repo every minute for changes, gitbuilder doesn't build every commit. It doesn't work like that, it checks out the master and builds that. If there is a failure it automatically bisects to find out where the problem first got introduced. Hope the change to the builder helps!
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnV2c63kDc6X21a1H81me1mIenUCScd2Gs"
 nickname="Emanuele"
 subject="watch branch?"
 date="2012-06-01T19:19:17Z"
 content="""
Hello there? Where can I find more info about this git watch branch?
Keep up the good work!
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnwfqF4wL6l_O26RyzoBowUMvQ_955Vpao"
 nickname="Markus"
 subject="comment 7"
 date="2014-06-14T06:29:55Z"
 content="""
FWIW: Removing the [raspi-copies-and-fills](https://github.com/simonjhall/copies-and-fills) package ('apt-get purge raspi-copies-and-fills') stops annex from printing the message over and over again. According to the package's description though, you suffer a performance penalty without the package.
"""]]


code:
Things that don't fit anywhere else:

* Automatically start daemon on boot or when user logs in, using
  freedesktop autostart file. **done**
* Somehow get content that is unavailable. This is problematic with inotify,
  since we only get an event once the user has tried (and failed) to read
  from the file. This is only needed if all the files in the directory
  are not kept synced, but in some situations (ie, low disk space phones),
  that is likely.
* Drop files that have not been used lately, or meet some other criteria
  (as long as there's a copy elsewhere). **done** (via preferred content;
  eg archive directories)
* Perhaps automatically dropunused files that have been deleted,
  although I cannot see a way to do that, since by the time the inotify
  deletion event arrives, the file is deleted, and we cannot see what
  its symlink pointed to! Alternatively, perhaps automatically
  do an expensive unused/dropunused cleanup process.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 1"
 date="2012-10-30T09:07:37Z"
 content="""
I think the automatic dropunused part is handled by the assistant already anyway, but in case it's not: The git repo still knows where the symlink pointed to so it should not be a problem to get that info from the last commit that was 'actively' locally and which contained said symlink.
"""]]


code:
Git-annex already supports storing large files in
several cloud providers via [[special_remotes]].
More should be added, such as:

* Google drive (attractive because it's free, only 5 gb tho)
* Owncloud (has several [providers](http://owncloud.org/providers/);
  at least one provides 5 gb free; open DAV based API)
* OpenStack Swift (teh future)
* Box.com (it's free, and current method is hard to set up and a sorta
  shakey; a better method would be to use its API) **done**
* Dropbox? That would be ironic.. Via its API, presumably.
* [[Amazon Glacier|todo/special_remote_for_amazon_glacier]] **done**
* Internet Archive **done**
* [nimbus.io](https://nimbus.io/) Fairly low prices ($0.06/GB);
  REST API; free software
* Mediafire provides 50gb free and has a REST API.
* Flickr provides 1 tb (!!!!) to free accounts, and can store at least
  photos and videos. <https://github.com/ricardobeat/filr> is a hack
  to allow storing any type of file on Flickr.
* mega.co.nz. Already supported via [[tips/megaannex]], would just need
  webapp modifications to configure it. May want to use megaannex as-is to
  build a non-hook special remote in haskell.

See poll at [[polls/prioritizing_special_remotes]].


code:
For git-annex to be able to clone its repo to another host, it'd be good to
have some way of pairing devices.

## security

Pairing uses its own network protocol, built on top of multicast UDP.

It's important that pairing securely verifies that the right host is being
paired with. This is accomplied by having a shared secret be entered on
both the hosts that will be paired. Hopefully that secret is communicated
securely out of band.

(In practice, the security of that communication will vary. To guard against
interception, each pairing session pairs exactly two hosts and then forgets
the shared secret. So an attacker who tries to reuse an intercepted secret
will not succeed in pairing. This does not guard against an intercepted
secret that is used before the legitimate parties finish pairing.)

Each host can construct messages that the other host can verify using the
shared secret, and so know that, for example, the ssh public key it
received belongs to the right host and has not been altered by a man in the
middle.

The verification works like this: Take a HMAC SHA1 checksum of the message,
using the shared secret as the HMAC key. Include this checksum after the
message. The other host can then do the same calculation and verify the
checksum.

Additionally, a UUID is included in the message. Messages that are part of
the same pairing session all share a UUID. And all such messages should
be verifiable as described above. If a message has the same UUID but is
not verifiable, then someone on the network is up to no good. Perhaps
they are trying to brute-force the shared secret. When this is detected,
the pairing session is shut down. (Which would still let an attacker
DOS pairing, but that's not a very interesting attack.)

The protocol used for pairing consists of 3 messages, a PairReq, and
PairAck, and a PairDone. Let's consider what an attacker could accomplish
by replaying these:

* PairReq: This would make the webapp pop up an alert about an incoming
  pair request. If the user thought it was real and for some reason
  entered the right shared secret used in the real one earlier, the
  ssh key inside the PairReq would be added to `authorized_keys`. Which
  allows the host that originally sent the PairReq to access its git
  repository, but doesn't seem to do the attacker any good.
* PairAck:  If the host that originally sent
  the PairReq is still pairing, it'll add the ssh key from the PairAck,
  and start syncing, which again does the attacker no good.
* PairDone: If the host that sent the PairAck is still syncing, it'll
  add the ssh key from the PairDone, and start syncing, and stop
  sending PairAcks. But probably, it's not syncing, because it would have
  seen the original PairDone.. and anyway, this seems to do the attacker no
  good.

So replay attacks don't seem to be a problem.

So far I've considered security from a third-party attacker, but either the
first or second parties in pairing could also be attackers. Presumably they
trust each other with access to their files as mediated by
[[git-annex-shell]]. However, one could try to get shell access to the
other's computer by sending malicious data in a pairing message. So the
pairing code always checks every data field's content, for example the ssh
public key is rejected if it looks at all unusual. Any control characters
in the pairing message cause it to be rejected, to guard against console
poisoning attacks. Furthermore, git-annex is careful not to expose data to
the shell, and the webapp uses Yesod's type safety to ensure all user input
is escaped before going to the browser.

## TODO

* pairing over IPV6 only networks does not work. Haskell's
  `network-multicast` library complains "inet_addr: Malformed address: ff02::1"
  .. seems it just doesn't support IPv6. The pairing code in git-annex
  does support ipv6, apart from this, it's just broadcasting the messages
  that fails. (Pairing over mixed networks is fine.)
* If there are three assistants on the network, and 2 pair, the third is
  left displaying a "Pair request from foo" alert, until it's close.
  Or, if the user clicks the button to pair, it'll get to the
  "Pairing in progress" alert, which will show forever (until canceled).

  It should be possible for third parties to tell when pairing is done,
  but it's actually rather hard since they don't necessarily share the secret.


code:
On a regular system, a reasonable simplifying assumption is that all the
files in the folder will be synced to the system. A user might want to
disable syncing of some subdirectories, for eg, archived files. But in
general, things are simpler to understand and implement if all files sync.

But, an Android gadget probably cannot hold all a user's files. Indeed,
it's likely that old files will be aggressively dropped from the Android
after syncing to elsewhere, in order to keep enough free space on it for
new files.

There needs to be a way for the user to browse files not on the gadget and
request they be transferred to it. This could be done as a browser in the
web app, or using a subdirectory full of placeholder files (not symlinks;
see [[Android]]) that start transfer of the real file when accessed.

----

Currently, Android uses the "source" repository type in some
configurations. This makes files be removed as soon as they are sent
somewhere else.

A compromise that avoids needing UI might be to change "source" so it
retained files for a while after they were created, even after they were
uploaded elsewhere. For example, it could hold onto them for a day. This
would allow the user time to do things with new files before they are
removed from the android device.

One way to implement that would be a new preferred content expression like
"age(1 day)". But this would need at least a daily full transfer scan to be
run.

Another way would be to have a way to make drops of files be deferred
for a period of time. This approach would not need to be specific to the
"source" repository type. And seems easy enough to do, just have a
configuration setting for the time interval, and an ordered drop queue
and a thread that waits as needed before dropping.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkpEY8WTFDhjIVTWG38Ph7ppmuXUTJAHAg"
 nickname="Justin"
 subject="selective sync"
 date="2012-07-28T04:08:00Z"
 content="""
hey joey

great work!!

will partial content work like selective sync in dropbox

use case: on desktop i have photos/mp3s/docs, but would only want to sync the docs to my netbook

cheers

justin
"""]]


code: [[!inline pages="(page(design/assistant/blog/*) and tagged(design/assistant/polls)) or page(design/assistant/polls/*)" show=0]]


code: The 2013 git-annex user survey is in progress [[here|/polls/2013]].


code:
Help me choose a goal for the month of December. The last poll showed
a lot of interest in using the git-annex assistant with phones, etc.

Background: git-annex uses symbolic links in its repositories. This makes it
hard to use with filesystems, such as FAT, that do not support symbolic links.
FAT filesystems are the main storage available on some Android devices that
have a micro-SD card. Other, newer Android devices don't have a SD card and so
avoid this problem.

I can either work on the idea described in
[[design/assistant/desymlink]], which could solve the symlink problem and
also could lead to a nicer workflow to editing files that are stored in
git-annex.

Or, I can work on [[Android_porting|design/assistant/android]], and try to
get the assistant working on Android's built-in storage.

[[!poll open=no 81 "solve the symlink problem first" 17 "port to Android first" 1 "other"]]