code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 2"
 date="2011-04-05T18:41:49Z"
 content="""
I see no use case for verifying encrypted object files w/o access to the encryption key. And possible use cases for not allowing anyone to verify your data.

If there are to be multiple encryption keys usable within a single encrypted remote, than they would need to be given some kind of name (a since symmetric key is used, there is no pubkey to provide a name), and the name encoded in the files stored in the remote. While certainly doable I'm not sold that adding a layer of indirection is worthwhile. It only seems it would be worthwhile if setting up a new encrypted remote was expensive to do. Perhaps that could be the case for some type of remote other than S3 buckets.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 3"
 date="2011-04-05T23:24:17Z"
 content="""
Assuming you're storing your encrypted annex with me and I with you, our regular cron jobs to verify all data will catch corruption in each other's annexes.

Checksums of the encrypted objects could be optional, mitigating any potential attack scenarios.

It's not only about the cost of setting up new remotes. It would also be a way to keep data in one annex while making it accessible only in a subset of them. For example, I might need some private letters at work, but I don't want my work machine to be able to access them all.
"""]]

repo_name: kakafarm/git

license: unknown


code:
[[!comment format=mdwn
 username="http://joey.kitenet.net/"
 nickname="joey"
 subject="comment 4"
 date="2011-04-07T19:59:30Z"
 content="""
@Richard the easy way to deal with that scenario is to set up a remote that work can access, and only put in it files work should be able to see. Needing to specify which key a file should be encrypted to when putting it in a remote that supported multiple keys would add another level of complexity which that avoids.

Of course, the right approach is probably to have a separate repository for work. If you don't trust it with seeing file contents, you probably also don't trust it with the contents of your git repository.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkS6aFVrEwOrDuQBTMXxtGHtueA69NS_jo"
 nickname="Hans"
 subject="using sshfs + cryptmount is more secure"
 date="2012-08-14T13:41:47Z"
 content="""
\"For git-annex, note that an attacker with local machine access can tell at least all the filenames and metadata of files stored in the encrypted remote anyway, and can access whatever content is stored locally.\"

Better security is given by sshfs + cryptmount, which I used when I recently setup a git-annex repository on a free shell account from a provider I do not trust.

See http://code.cjb.net/free-secure-online-backup.html for what I did to get a really secure solution.

Kind regards,

Hans Ekbrand
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo"
 nickname="Justin"
 subject="comment 6"
 date="2012-08-14T14:10:40Z"
 content="""
Hans,

You are misunderstanding how git-annex encryption works.  The \"untrusted host\" and the \"local machine\" are not the same machine.  git-annex only transfers pre-encrypted files to the \"untrusted host\".

You should setup a git-annex encrypted remote and watch how it works so you can see for yourself that it is not insecure.

Your solution does not provide better security, it accomplishes the same thing as git-annex in a more complicated way.  In addition, since you are mounting the image from the client your solution will not work with multiple clients.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkS6aFVrEwOrDuQBTMXxtGHtueA69NS_jo"
 nickname="Hans"
 subject="comment 7"
 date="2012-08-15T19:16:10Z"
 content="""
Justin,

thanks for clearing that up. It's great that git-annex has implemented mechanisms to work securely on untrusted hosts. My solution is thus only interesting for files that are impractical to manage with git-annex (e.g. data for/from applications that need rw-access to a large number of files). And, possibly, for providers that do not provide rsync.

Your remark that my solution does not work with more than one client, is not entirely accurate. No more than one client can access the repository at any given time, but as long as access is not simultaneous, any number of clients can access the repository. Still, your point is taken, it's a limitation I should mention.

It would be interesting to compare the performance of individually encrypted files to encrypted image-file. My intuition says that encrypted image-file should be faster, but that's just a guess.
"""]]


code:
Communication between git-annex and a program implementing an external
special remote uses this protocol.

[[!toc]]

## starting the program

The external special remote program has a name like
`git-annex-remote-$bar`. When
`git annex initremote foo type=external externaltype=$bar` is run,
git-annex finds the appropriate program in PATH.

The program is started by git-annex when it needs to access the special
remote, and may be left running for a long period of time. This allows
it to perform expensive setup tasks, etc. Note that git-annex may choose to
start multiple instances of the program (eg, when multiple git-annex
commands are run concurrently in a repository).

## protocol overview

Communication is via stdin and stdout. Therefore, the external special
remote must avoid doing any prompting, or outputting anything like eg,
progress to stdout. (Such stuff can be sent to stderr instead.)

The protocol is line based. Messages are sent in either direction, from
git-annex to the special remote, and from the special remote to git-annex.

In order to avoid confusing interactions, one or the other has control
at any given time, and is responsible for sending requests, while the other
only sends replies to the requests.

Each protocol line starts with a command, which is followed by the
command's parameters (a fixed number per command), each separated by a
single space. The last parameter may contain spaces. Parameters may be
empty, but the separating spaces are still required in that case.

## example session

The special remote is responsible for sending the first message, indicating
the version of the protocol it is using.

	VERSION 1

Once it knows the version, git-annex will generally
send a message telling the special remote to start up.
(Or it might send a INITREMOTE, so don't hardcode this order.)

	PREPARE

The special remote can now ask git-annex for its configuration, as needed,
and check that it's valid. git-annex responds with the configuration values

	GETCONFIG directory
	VALUE /media/usbdrive/repo
	GETCONFIG automount
	VALUE true

Once the special remote is satisfied with its configuration and is
ready to go, it tells git-annex.

	PREPARE-SUCCESS

Now git-annex will tell the special remote what to do. Let's suppose
it wants to store a key.

	TRANSFER STORE somekey tmpfile

The special remote can continue sending messages to git-annex during this
transfer. It will typically send progress messages, indicating how many
bytes have been sent:

	PROGRESS 10240
	PROGRESS 20480

Once the key has been stored, the special remote tells git-annex the result:

	TRANSFER-SUCCESS STORE somekey

Once git-annex is done with the special remote, it will close its stdin.
The special remote program can then exit.

## git-annex request messages

These are messages git-annex sends to the special remote program.
None of these messages require an immediate reply. The special
remote can send any messages it likes while handling the requests.

Once the special remote has finished performing the request, it should
send one of the corresponding replies listed in the next section.

The following requests *must* all be supported by the special remote.

* `INITREMOTE`
  Request that the remote initialize itself. This is where any one-time
  setup tasks can be done, for example creating an Amazon S3 bucket.
  Note: This may be run repeatedly over time, as a remote is initialized in
  different repositories, or as the configuration of a remote is changed.
  (Both `git annex initremote` and `git-annex enableremote` run this.)
  So any one-time setup tasks should be done idempotently.
* `PREPARE`
  Tells the special remote it's time to prepare itself to be used.
  Only INITREMOTE can come before this.
* `TRANSFER STORE|RETRIEVE Key File`
  Requests the transfer of a key. For STORE, the File is the file to upload;
  for RETRIEVE the File is where to store the download.
  Note that the File should not influence the filename used on the remote.
  The filename will not contain any whitespace.
  Note that it's important that, while a Key is being stored, CHECKPRESENT
  not indicate it's present until all the data has been transferred.
  Multiple transfers might be requested by git-annex, but it's fine for the
  program to serialize them and only do one at a time.
* `CHECKPRESENT Key`
  Requests the remote check if a key is present in it.
* `REMOVE Key`
  Requests the remote remove a key's contents.

The following requests can optionally be supported. If not handled,
replying with `UNSUPPORTED-REQUEST` is acceptable.

* `GETCOST`
  Requests the remote return a use cost. Higher costs are more expensive.
  (See Config/Cost.hs for some standard costs.)
* `GETAVAILABILITY`
  Requests the remote send back an `AVAILABILITY` reply.
  If the remote replies with `UNSUPPORTED-REQUEST`, its availability
  is asssumed to be global. So, only remotes that are only reachable
  locally need to worry about implementing this.

More optional requests may be added, without changing the protocol version,
so if an unknown request is seen, reply with `UNSUPPORTED-REQUEST`.

## special remote replies

These should be sent only in response to the git-annex request messages.
They do not have to be sent immediately after the request; the special
remote can send its own requests (listed in the next section below)
while it's handling a request.

* `PREPARE-SUCCESS`
  Sent as a response to PREPARE once the special remote is ready for use.
* `PREPARE-FAILURE ErrorMsg`
  Sent as a response to PREPARE if the special remote cannot be used.
* `TRANSFER-SUCCESS STORE|RETRIEVE Key`
  Indicates the transfer completed successfully.
* `TRANSFER-FAILURE STORE|RETRIEVE Key ErrorMsg`
  Indicates the transfer failed.
* `CHECKPRESENT-SUCCESS Key`
  Indicates that a key has been positively verified to be present in the
  remote.
* `CHECKPRESENT-FAILURE Key`
  Indicates that a key has been positively verified to not be present in the
  remote.
* `CHECKPRESENT-UNKNOWN Key ErrorMsg`
  Indicates that it is not currently possible to verify if the key is
  present in the remote. (Perhaps the remote cannot be contacted.)
* `REMOVE-SUCCESS Key`
  Indicates the key has been removed from the remote. May be returned if
  the remote didn't have the key at the point removal was requested.
* `REMOVE-FAILURE Key ErrorMsg`
  Indicates that the key was unable to be removed from the remote.
* `COST Int`
  Indicates the cost of the remote.
* `AVAILABILITY GLOBAL|LOCAL`
  Indicates if the remote is globally or only locally available.
  (Ie stored in the cloud vs on a local disk.)
* `INITREMOTE-SUCCESS`
  Indicates the INITREMOTE succeeded and the remote is ready to use.
* `INITREMOTE-FAILURE ErrorMsg`
  Indicates that INITREMOTE failed.
* `UNSUPPORTED-REQUEST`
  Indicates that the special remote does not know how to handle a request.

## special remote messages

These messages may be sent by the special remote at any time that it's
in control.

* `VERSION Int`
  Supported protocol version. Current version is 1. Must be sent first
  thing at startup, as until it sees this git-annex does not know how to
  talk with the special remote program!
* `PROGRESS Int`
  Indicates the current progress of the transfer (in bytes). May be repeated
  any number of times during the transfer process, but it's wasteful to
  update the progress until at least another 1% of the file has been sent.
  This is highly recommended for STORE. (It is optional but good for RETRIEVE.)
  (git-annex does not send a reply to this message.)
* `DIRHASH Key`
  Gets a two level hash associated with a Key. Something like "abc/def".
  This is always the same for any given Key, so can be used for eg,
  creating hash directory structures to store Keys in.
  (git-annex replies with VALUE followed by the value.)
* `SETCONFIG Setting Value`
  Sets one of the special remote's configuration settings.
  Normally this is sent during INITREMOTE, which allows these settings
  to be stored in the git-annex branch, so will be available if the same
  special remote is used elsewhere. (If sent after INITREMOTE, the changed
  configuration will only be available while the remote is running.)
* `GETCONFIG Setting`
  Gets one of the special remote's configuration settings, which can have
  been passed by the user when running `git annex initremote`, or
  can have been set by a previous SETCONFIG. Can be run at any time.
  (git-annex replies with VALUE followed by the value. If the setting is
  not set, the value will be empty.)
* `SETCREDS Setting User Password`
  When some form of user and password is needed to access a special remote,
  this can be used to securely store them for later use.
  (Like SETCONFIG, this is normally sent only during INITREMOTE.)
  The Setting indicates which value in a remote's configuration can be
  used to store the creds.
  Note that creds are normally only stored in the remote's configuration
  when it's surely safe to do so; when gpg encryption is used, in which
  case the creds will be encrypted using it. If creds are not stored in
  the configuration, they'll only be stored in a local file.
  (embedcreds can be set to yes by the user or by SETCONFIG to force
   the creds to be stored in the remote's configuration).
* `GETCREDS Setting`
  Gets any creds that were previously stored in the remote's configuration
  or a file.
  (git-annex replies with "CREDS User Password". If no creds are found,
  User and Password are both empty.)
* `GETUUID`
  Queries for the UUID of the special remote being used.
  (git-annex replies with VALUE followed by the UUID.)
* `GETGITDIR`
  Queries for the path to the git directory of the repository that
  is using the external special remote.
  (git-annex replies with VALUE followed by the path.)
* `SETWANTED PreferredContentExpression`
  Can be used to set the preferred content of a repository. Normally
  this is not configured by a special remote, but it may make sense
  in some situations to hint at the kind of content that should be stored
  in the special remote. Note that if a unparsable expression is set,
  git-annex will ignore it.
* `GETWANTED`
  Gets the current preferred content setting of the repository.
  (git-annex replies with VALUE followed by the preferred content
  expression.)
* `SETSTATE Key Value`
  Can be used to store some form of state for a Key. The state stored
  can be anything this remote needs to store, in any format.
  It is stored in the git-annex branch. Note that this means that if
  multiple repositories are using the same special remote, and store
  different state, whichever one stored the state last will win. Also,
  it's best to avoid storing much state, since this will bloat the
  git-annex branch. Most remotes will not need to store any state.
* `GETSTATE Key`
  Gets any state that has been stored for the key.
  (git-annex replies with VALUE followed by the state.)
* `DEBUG message`
  Tells git-annex to display the message if --debug is enabled.

## general messages

These messages can be sent at any time by either git-annex or the special
remote.

* `ERROR ErrorMsg`
  Generic error. Can be sent at any time if things get too messed up
  to continue. When possible, use a more specific reply from the list above.
  The special remote program should exit after sending this, as
  git-annex will not talk to it any further. If the program receives
  an ERROR from git-annex, it can exit with its own ERROR.

## long running network connections

Since an external special remote is started only when git-annex needs to
access the remote, and then left running, it's ok to open a network
connection in the PREPARE stage, and continue to use that network
connection as requests are made.

If you're unable to open a network connection, or the connection closes,
perhaps because the network is down, it's ok to fail to perform any
requests. Or you can try to reconnect when a new request is made.

Note that the external special remote program may be left running for
quite a long time, especially when the git-annex assistant is using it.
The assistant will detect when the system connects to a network, and will
start a new process the next time it needs to use a remote.

## TODO

* When storing encrypted files stream the file up/down the pipe, rather
  than using a temp file. Will probably involve less space and disk IO,
  and makes the progress display better, since the encryption can happen
  concurrently with the transfer. Also, no need to use PROGRESS in this
  scenario, since git-annex can see how much data it has sent/received from
  the remote. However, \n and probably \0 need to be escaped somehow in the
  file data, which adds complication.
* uuid discovery during INITREMOTE.
* Support for getting and setting the list of urls that can be associated
  with a key.
* Hook into webapp. Needs a way to provide some kind of prompt to the user
  in the webapp, etc.


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.43"
 subject="comment 11"
 date="2014-01-13T18:45:34Z"
 content="""
I've handled the cloud repo check by making external special remotes be assumed by default to be globally available via the cloud. So no need to do anything in most cases. For remotes that are only available locally, the remote can reply with \"AVAILABILITY LOCAL\" when git-annex sends an AVAILABILITY request.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8"
 nickname="TobiasTheViking"
 subject="Feature request"
 date="2014-01-11T15:41:48Z"
 content="""
The ability to mark a remote as being a \"cloud\" remote. To silence the \"Unable to download files from your other devices. Add a cloud repository\" message in the webapp.

Maybe as simple as \"SETCONFIG cloud true\", if that is a viable implementation.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm_YXzEdPHzbSGVwtmTR7g1BqDtTnIBB5s"
 nickname="Matthias"
 subject="Chunk it"
 date="2014-01-20T16:22:09Z"
 content="""
> TODO: stream the file up/down the pipe, rather than using a temp file

You might want to use chunked transfer, i.e. a series of \"EXPECT 65536\" followed by that many bytes of binary data and an EOF marker (EXPECT-END or EXPECT 0), instead of escaping three characters (newline, NUL, and the escape prefix) and the additional unnecessary tedious per-character processing that would require.
"""]]


code:
[[!comment format=mdwn
 username="http://schnouki.net/"
 nickname="Schnouki"
 subject="comment 13"
 date="2014-02-10T18:22:55Z"
 content="""
First things first: in the documentation, I think `SETCONFIG Setting` should be `SETCONFIG Setting Value`.

Now a few questions:

- why have `SETCREDS` and `GETCREDS` have both a username and a password? I'd like to use them to store a OAuth token, but because of this I also have to store a dummy value, which seems weird to me. Is it possible to just do `SETCREDS oauth_token XXXYYY123456`?
- about `PROGRESS`: my remote is sending `PROGRESS xxx` every 64kb uploaded or downloaded, but no upload/download progress is displayed by git-annex. Is this normal? Should I do it myself, or will it be done by a future version of git-annex?

Joey, thanks a lot for all your work on git-annex :)
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.172"
 subject="comment 14"
 date="2014-02-11T01:36:51Z"
 content="""
Schnouki, fixed SETCONFIG docs. Note that this is a wiki. :)

I agree it's a little weird for SETCREDS to have a username and a password. This is just exposing git-annex's existing credential storage which has a tuple of values rather than using, say, a multivalue map. If you only need one it's fine to put in a dummy value for the other one.

Re PROGRESS, it seems I hooked up the progress stuff, so it was visible in the webapp, but forgot to put up a progress display at the command line. Fixed in git.

I look forward to seeing your special remote implementation!
"""]]


code:
[[!comment format=mdwn
 username="http://schnouki.net/"
 nickname="Schnouki"
 subject="comment 15"
 date="2014-02-11T13:44:10Z"
 content="""
Joey, thanks for fixing that so quickly. Indeed it works in the webapp; I'll check the CLI version as soon as possible :)

I just released the new remote on <https://github.com/Schnouki/git-annex-remote-hubic>. It's for [hubiC](https://hubic.com/), a French personal cloud storage service made by OVH that has free 25GB accounts and only charges €10/month for 10TB (VAT included). It's really experimental (hacked it over the weekend), but it seems to work for me so far.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmicVKRM8vJX4wPuAwlLEoS2cjmFXQkjkE"
 nickname="Thomas"
 subject="not useful for "plain directory" special remote?"
 date="2013-12-16T20:10:16Z"
 content="""
It says: \"Note that the File should not influence the filename used on the remote. The filename used should be derived from the Key.\"

Thus this interface might not be useful to implement [[todo/New special remote suggeston - clean directory]]? The clean directory special remote would just do that: save $key content on the remote under the filename $file.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.87"
 subject="comment 2"
 date="2013-12-16T20:42:23Z"
 content="""
That's not how special remotes work -- they have nothing to do with the symlinks in the work tree, which are managed by git (or git-annex in direct mode). They only store values for keys.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8"
 nickname="TobiasTheViking"
 subject="Feature requests"
 date="2013-12-28T13:57:35Z"
 content="""
PREPARE-Failure ErrorMsg (matching INITREMOTE-FAILURE ErrorMsg)

Also, i'd like for the following to overwrite existing credentials/configs
MYFOLDER=\"testfolder\" MYLOGIN=\"login\" MYPASSWORD=\"pword\" MYURL=\"http://webdav/\"    git annex enableremote owncloud type=external externaltype=owncloud --debug

This would also be needed for refreshing oauth.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 4"
 date="2013-12-29T17:50:23Z"
 content="""
Added PREPARE-FAILURE

git-annex enableremote causes INITREMOTE to be called, so any credentials can be stored etc. (Note that, as with built-in special remotes, credentials are only stored in the git-annex branch when the remote is encrypted. Otherwise, they're stored locally in a .git/annex/creds/ file.)

Also, I'd recommend using environment variables for passing credentials to initremote/enableremote, because that avoids leaking them in `ps`.. but it probably doesn't make sense to use environment variables for other settings, but instead pass them as parameters of initremote/enableremote, which can be looked up using GETCONFIG. Only exception might be if the setting needs to vary between different machines.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8"
 nickname="TobiasTheViking"
 subject="Feature requests"
 date="2013-12-31T14:05:07Z"
 content="""
Hook should be able to set default configuration for itself.

For instance, clean flickr hook will only upload some files(notably pictures). The user shouldn't have to manage that.

Other hooks have a maximum filesize(though i guess that doesn't matter once splitting works).

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 6"
 date="2013-12-31T17:56:34Z"
 content="""
I suppose you're talking about preferred content settings.

I think that it makes sense for hooks to use existing git-annex plumbing etc when it's available. So a hook could just run `git annex wanted` to manage its preferred content.

The only problem is that a hook does not currently have a way to discover the UUID of the repository! So I've added a GETUUID to cover this and other use cases.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8"
 nickname="TobiasTheViking"
 subject="comment 7"
 date="2013-12-31T18:20:32Z"
 content="""
I think the hook running anything in shell, to interact with git annex, is a mistake.

I see a lot more potential pitfalls and mistakes(especially crossplatform).

It should be the existing git annex plumbing (preferred content) as you say. I just really think it should be configurable in the protocol, instead of a having to run a shell command.

Since you have made this advanced protocol i really see it as a mistake to do anything between the hook and git-annex outside of the protocol, it makes much more sense to have all their interactions happen within the protocol.

IMO anyways.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 8"
 date="2013-12-31T19:16:16Z"
 content="""
It makes sense to only implement one interface to things, unless there is a reason such as performance to do otherwise.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.227"
 subject="comment 9"
 date="2014-01-02T00:15:28Z"
 content="""
Tobias made some good points:

* git-annex may not be in PATH depending on installation method
* It would in theory be bad if a special remote ran some git-annex command that used the special remote and ran some git-annex command [...].
* git-annex would need to tell the special remote what git repo it was being used with.

So, added GETWANTED and SETWANTED. However, if I find myself recapitulating a lot of git-annex's command-line plumbing stuff in this protocol, I will need to revisit this decision and find a better way. Particularly, I narrowly escaped an intractable dependency loop in [[!commit 8e3032df2d5c6ddf07e43de4b3bb89cb578ae048]].
"""]]


code:
To integrate with git-remote-gcrypt, a key thing is to have a way to map
from the gcrypt-id of an encrypted repository to a git-annex repository
uuid.

To do this, we'll make a v5 UUID, feeding in the gcrypt-id.
The namespace used is itself a v5 UUID, generated using the URL
namespace and the URL of this page at the time this scheme was
developed: "http://git-annex.branchable.com/design/gcrypt/"


code:
# goals

* be configured like a regular git remote, with an unusual url
  or other configuration
* receive notifications when a remote has received new commits,
  and take some action
* optionally, do receive-pack and send-pack to a remote that
  is only accessible over an arbitrary network transport
  (like assistant does with XMPP)
* optionally, send/receive git-annex objects to remote
  over an arbitrary network transport

# difficulties

* authentication & configuration
* multiple nodes may be accessible over a single network transport,
  with it desirable to sync with any/all of them. For example, with
  XMPP, there can be multiple friends synced with. This means that
  one git remote can map to multiple remote nodes. Specific to git-annex,
  this means that a set of UUIDs known to be associated with the remote
  needs to be maintained, while currently each remote can only have one
  annex-uuid in .git/config.

# payoffs

* support [[assistant/telehash]]!
* Allow running against a normal ssh git remote. This would run
  git-annex-shell on the remote, watching for changes, and so be able to
  notify when a commit was pushed to the remote repo. This would let the
  assistant immediately notice and pull. So the assistant would be fully
  usable with a single ssh remote and no other configuration!
  **do this first**
* clean up existing XMPP support, make it not a special case, and not
  tightly tied to the assistant
* git-remote-daemon could be used independantly of git-annex,
  in any git repository.

# design

Let git-remote-daemon be the name. Or for git-annex,
`git annex remotedaemon`.

It runs in one of two ways:

1. Forked to background, using a named pipe for the control protocol.
2. With --foreground, the control protocol goes over stdio.

Either way, behavior is the same:

* Get a list of remotes to act on by looking at .git/config
* Automatically notices when a remote has changes to branches
  matching remote.$name.fetch, and pulls them down to the appropriate
  location.
* When the control protocol informs it about a new ref that's available,
  it offers the ref to any interested remotes.

# control protocol

This is an asynchronous protocol. Ie, either side can send any message
at any time, and the other side does not send a reply.

It is line based and intended to be low volume and not used for large data.

TODO: Expand with commands for sending/receiving git-annex objects, and
progress during transfer.

TODO: Will probably need to add something for whatever pairing is done by
the webapp.

## emitted messages

* `CONNECTED uri`

  Sent when a connection has been made with a remote.

* `DISCONNECTED uri`

  Sent when connection with a remote has been lost.

* `SYNCING uri`

  Indicates that a pull or a push with a remote is in progress.
  Always followed by DONESYNCING.

* `DONESYNCING uri 1|0`

  Indicates that syncing with a remote is done, and either succeeded
  (1) or failed (0).

* `WARNING uri string`

  A message to display to the user about a remote.

## consumed messages

* `PAUSE`

  The user has requested a pause.
  git-remote-daemon should close connections and idle.

* `LOSTNET`

  The network connection has been lost.
  git-remote-daemon should close connections and idle.

* `RESUME`

  Undoes PAUSE or LOSTNET.
  Start back up network connections.

* `CHANGED ref ...`

  Indicates that a ref is new or has changed. These can be offered to peers,
  and peers that are interested in them can pull the content.

* `RELOAD`

  Indicates that configs have changed. Daemon should reload .git/config
  and/or restart.

  Possible config changes include adding a new remote, removing a remote,
  or setting `remote.<name>.annex-sync` to configure whether to sync with a
  particular remote.

* `STOP`

  Shut down git-remote-daemon

  (When using stdio, it also should shutdown when it reaches EOF on
  stdin.)

# encryption & authentication

For simplicity, the network transports have to do their own end-to-end
encryption. Encryption is not part of this design.

(XMPP does not do end-to-end encryption, but might be supported
transitionally.)

Ditto for authentication that we're talking to who we intend to talk to.
Any public key data etc used for authentication is part of the remote's
configuration (or hidden away in a secure chmodded file, if necessary).
This design does not concern itself with authenticating the remote node,
it just takes the auth token and uses it.

For example, in telehash, each node has its own keypair, which is used
or authentication and encryption, and is all that's needed to route
messages to that node.

# network level protocol

How do peers communicate with one another over the network?

This seems to need to be network-layer dependant. Telehash will need
one design, and git-annex-shell on a central ssh server has a very different
(and much simpler) design.

## ssh

`git-annex-shell notifychanges` is run, and speaks a simple protocol
over stdio to inform when refs on the remote have changed.

No pushing is done for CHANGED, since git handles ssh natively.

TODO: test!

## telehash

TODO

## xmpp

Reuse [[assistant/xmpp]]


code:
[[!comment format=mdwn
 username="http://johan.kiviniemi.name/"
 nickname="Johan"
 subject="Rolling hash chunking"
 date="2014-04-04T14:16:25Z"
 content="""
I am not sure which page is the best for this comment, but this one seems somewhat relevant.

Given that a future telehash implementation may download files from multiple peers, it might be a good idea to download files in chunks, possibly in parallel. In this case, it might be a good idea to use a rolling hash for chunking (like rsync et al). [There is a package for that on Hackage](http://hackage.haskell.org/package/hash-0.2.0.1/docs/Data-Hash-Rolling.html).

git-annex could store a list of chunk checksums in `.git/annex/objects/…/SHA….chunks` whenever the repository holds a copy of the file. The checksum list would be a small fraction of the file in size, but all the checksum lists for all the files in a repository might take up too much space to store in the `git-annex` branch.

When getting an object, git-annex could first download the `.chunks` file from a remote/peer and then proceed to download missing chunks in a BitTorrent-like fashion.

If git-annex has an idea about what locally present object might be an earlier version of the file, it could compare the checksum lists and only download the parts that have changed (à la rsync).
"""]]


code:
[[!toc]]

# metadata

Attach an arbitrary set of metadata to a key. This consists of any number
of fields. Each field has an unordered set of values. The special field
"tag" has as its values any tags that are set for the key.

Store in git-annex branch, next to location log files.

Storage needs to support union merging, including removing an old value
of a field, and adding a new value of a field.

# filtered branches

See [[tips/metadata_driven_views]]

The reason to use specially named filtered branches is because it makes
self-documenting how the repository is currently filtered.

## unmatched files in filtered branches

TODO Files not matching the view should be able to be included in
the filtered branch, in a special location, an "other" directory.

For example, it could make a "other" directory containing files
without a tag when viewing by tag.

It might be nice, if in a two level view, for the other directories
to nest. For example, `other/2014/file`. However, that leads to a
performance problem: When adding a level to a view, it has to look at each
file in the "other" directory and generate a view for it too. With a lot
of files, that'd be slow.

Instead, why not replicate the parent branch's directory structure inside
the "other" directory? Then the directory tree only has to be constructed
once, and can be left alone when refining a view.

## operations while on filtered branch

* If files are removed and git commit called, git-annex should remove the
  relevant metadata from the files. **done**
  (Currently, only metadata used for visible subdirs is added and removed
  this way.)
  (Also, this is not usable in direct mode because deleting the
  file.. actually deletes it...)
* If a file is moved into a new subdirectory while in a view branch,
  a tag is added with the subdir name. This allows on the fly tagging.
  **done**
* `git annex sync` should avoid pushing out the view branch, but
  it should check if there are changes to the metadata pulled in, and update
  the branch to reflect them.

## automatically added metadata

When annex.genmetadata is set, git annex add automatically attaches
some metadata to a file. Currently year and month fields, from its mtime.

There's also a post-commit-annex hook script.

## directory hierarchy metadata

From the original filename used in the master branch, when
constructing a view, generate fields. For example foo/bar/baz.mp3
would get /=foo, foo/=bar, foo/bar/=baz, and .=mp3.

Note that dir/=subdir allows a view to use `dir/=*` and only
match one level of subdirs with the glob. So is better than dir=foo/bar
as the metadata. (Alternatively, could do special glob matching.)

This allows using whatever directory hierarchy exists to inform the view,
without locking the view into using it.

Complication: When refining a view, it only looks at the filenames in
the view, so it has to map from
those filenames to derive the same metadata, unless there is persistent
storage. Luckily, the filenames used in the views currently include the
subdirs.

# other uses for metadata

Uses are not limited to view branches.

`git annex checkoutmeta year=2014 talk` in a subdir of master could create the
same tree of files filter would. The user can then commit that if desired.
Or, they could run additional commands like `git annex fadd` to refine the
tree of files in the subdir.

Metadata can be used for configuring numcopies. One way would be a
numcopies=n value attached to a file. But perhaps better would be to make
the numcopies.log allow configuring numcopies based on which files have
other metadata.

Other programs could query git-annex for the metadata of files in the work
tree, and do whatever it wants with it.

# filenames

The hard part of this is actually getting a useful filename to put in the
view branch, since git-annex only has a key which the user will not
want to see.

* Could use filename metadata for the key, recorded by git-annex add (which
  may not correspond to filenames being used in regular git branches like
  master for the key).
* Could use the .map files to get a filename, but this is somewhat
  arbitrary (.map can contain multiple filenames), and is only
  currently supported in direct mode.
* Current approach: Have a reference branch (eg master) and walk it to
  find filenames and
  keys. Fine as long as it can be done efficiently. Also allows including
  the subdirectory a file is in, potentially. cwebber points out that this
  is essentially a form of tracking branch. Which implies it will need to
  be updatable when the reference branch changes. Should be doable via
  diff-tree.

Note that we have to take care to avoid generating conflicting filenames.
The current approach is to embed the full directory structure inside the
filename in the view branch.

## union merge properties

While the storage could just list all the current values of a field on a
line with a timestamp, that's not good enough. Two disconnected
repositories can make changes to the values of a field (setting and
unsetting tags for example) and when this is union merged back together,
the changes need to be able to be replayed in order to determine which
values we end up with.

To make that work, we log not only when a field is set to a value,
but when a value is unset as well.

For example, here two different remotes added tags, and then later
a tag was removed:

	1287290776.765152s tag +foo +bar
	1287290991.152124s tag +baz
	1291237510.141453s tag -bar

# efficient metadata lookup

Looking up metadata for view generation so far requires traversing all keys
in the git-annex branch. This is slow. A fast cache is needed.

TODO

# direct mode issues

TODO (direct mode is currently not supported with view branches)

Checking out a view branch can result in any number of copies of a file
appearing in different directories. No problem in indirect mode, but
in direct mode these are real, expensive copies.

But, it's worth supporting direct mode!

So, possible approaches:

* Before checking out a view branch, calculate how much space will
  be used by duplicates and refuse if not enough is free.
* Only check out one file, and omit the copies. Keep track of which
  files were omitted, and make sure that when committing on the branch,
  that metadata is not removed. Has the downside that files can seem
  to randomly move around in the tree as their metadata changes.
* Disallow view branch checkouts that have duplicate files.
  This would cripple it some, but perhaps not too badly?

# gotchas

* Checking out a view branch can remove the current subdir. May be worth
  detecting when this happens and help the user.
  **done**

* Git has a complex set of rules for what is legal in a ref name.
  View branch names will need to filter out any illegal stuff. **done**

* Metadata should be copied to the new key when adding a modified version
  of a file. **done**

* Filesystems that are not case sensative (including case preserving OSX)
  will cause problems if view branches try to use different cases for
  2 directories representing a metadata field.

  Solution might be to compare fields names case-insensitively, and
  pick one representation consistently. **done**

* Assistant needs to know about views, so it can update metadata when
  files are moved around inside them. TODO

* What happens if git annex add or the assistant add a new file while on a
  view? If the file is not also added to the master branch, it will be lost
  when exiting the view. TODO

* The filename mangling can result in a filename in a view
  that is too long for its containing filesystem. Should detect and do
  something reasonable to avoid. TODO


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm3vKzS4eOWYpKMoYXqMIjNsIg_nYF-loU"
 nickname="Konubinix"
 subject="Already existing metadata implementation "
 date="2014-02-22T21:45:25Z"
 content="""
Hi,

I love the idea behing storing metadata.

I suggest to exchange ideas (and maybe code) with projects already implementing metadata systems.

I have tried several implementations and particularly noticed tmsu (http://tmsu.org/). This tool stores tags into a sqlite database and uses also a SHA-256 fingerprint of the file to be aware of file moves. It provides a fuse view of the tags with the ability to change tags by moving files (like in the git annex metadata view).

Paul Ruane is particularly responsive on the mailing list and he already supports git annexed files (with SHAE-256 fingerprint) (see the end of the thread https://groups.google.com/forum/#!topic/tmsu/A5EGpnCcJ2w).

Even if you cannot reuse the project, they are interresting ideas that might be worth looking at like the implications of tags: a file tagged \"film\" being automatically tagged \"video\".

Tagsistant (http://www.tagsistant.net/) may also be a good source of inspirations. I just don't like the fact that it uses a backstore of tagged files.

Thanks for reading.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 2"
 date="2014-02-25T09:51:17Z"
 content="""
Some additional ideas for metadata...

Instead of having a simplistic scheme like 'field=value' it might be advantageous to consider a scheme like 'attribute=XXX, value=YYY, unit=ZZZ' that way you could do intesting things with the metadata like adding counters to things, and allow for doing interesting queries like give me all 'things' tagged with a unit of \"audio_file\", this assumes one had trawled through an entire annex and then tagged all files based on type with the unix file tool or something like that.

The above idea is already in use in irods and its a really nice and powerful way to let users add meta-data and to build up more interesting use cases and tools.

btw, I plan on taking a look at seeing if I can map some of the meta that we have in work into this new git-annex feature to see how well/bad it works. Either way this feature looks cool! +1!!!
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 3"
 date="2014-02-25T09:57:09Z"
 content="""
actually in your mp3 example you could have ....

ATTRIBUTE=sample_rate, VALUE=22100, UNIT=Hertz

another example use case is to always be consistent with the AVU order then you could stick in ntriples from RDF to do other cool things by looking up various linked data sources -- see http://www.w3.org/2001/sw/RDFCore/ntriples/ and http://www.freebase.com/, actually this would be quite cool if git-annex examined the mp3's id3 tag, the created an ntriple styled entry can be automatically parsed with the web-based annex gui and automatically pull in additional meta-data from the likes of freebase. I guess the list of ideas can just only get bigger with this potential metadata capability.
"""]]


code:
[[!comment format=mdwn
 username="bremner"
 ip="198.164.160.48"
 subject="convenient way to query metadata?"
 date="2014-03-15T20:58:28Z"
 content="""
I'd like to be able to do something like \"git annex metadata -q fieldname\" and have that output the value(s) of fieldname.  I see I could parse the json output but that isn't too convenient in a shell script. Or have I missed something that already exists?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.154"
 subject="comment 5"
 date="2014-03-15T21:30:52Z"
 content="""
@bremner, you must be up to something interesting.. Added metadata --get for you.
"""]]


code:
[[!comment format=mdwn
 username="tdussa"
 ip="217.84.74.69"
 subject="Why not automatically add the whole date?"
 date="2014-04-30T20:41:20Z"
 content="""
Hi,

apologies if I am missing something, but from what I understand, git-annex will automatically add the year and the month from a file's mtime to its metadata if instructed to do so.

So... What about the day (or the time, for that matter?)?  What is the reasoning behind the decision not to add those bits automatically?  And, is there a way to get git-annex to add those bits of information automatically as well (besides the obvious way of creating a pre-commit-hook script to that effect)?

THX & Cheers,
Toby.
"""]]


code:
The [[preferred_content]] expressions didn't have a design document, but
it's a small non-turing complete DSL for expressing which objects a
repository prefers to contain.

One thing that needs to be written down though is the stability analysis
that must be done of preferred content expressions.

It's important that when a set of repositories all look at one-another's
preferred content expressions, and copy/move/drop objects to satisfy them,
they end up at a steady state. So, a given preferred content expression
should ideally evaluate to the same answer for each key, from the
perspective of each repository.

The best way to ensure that is the case is to only use terms in preferred
content expressions that rely on state that is shared between all
repositories. So, state in the git-annex branch, or the master branch
(assuming all repositories have master checked out).

Since git is eventually consistent, there might be disagreements about
which object belongs where, but once consistency is reached, things will
settle down.


code:
## requesting content

In some situations, nodes only want particular files, and not everything.
(Or don't have the bandwidth to get everything.) A way to handle this,
that should work in a fully ad-hoc, offline distributed network,
suggested by Vincenzo Tozzi:

* Nodes generate a request for a specific file they want, committed
  to git somewhere.
* This request has a TTL (of eg 3 or 4).
* When syncing, copy the requests that a node has, and decrease their TTL
  by 1. Requests with a TTL of 0 have timed out and are not copied.
  (So, requests are stored in git, but on eg, per-node branches.)
* Only copy content to nodes that have a request for it (either one
  originating with them, or one they copied from another node).
* Each request indicates the requesting node, so once no nodes have an
  active request for a particular file, it's ok to drop it from the
  transfer nodes (honoring numcopies etc of course).

## simulation

A simulation of a network using this method is in [[simroutes.hs]].

Question: How efficient is this method? Does the network fill with many
copies that are not needed, before the request is fulfilled?

## storing requests

Requests could be stored in the location tracking file.

Currently:

	time 0|1 uuid1
	time 0|1 uuid2

* Use negative numbers for the TTL of a request:

	time -3! uuid1
	time -2 uuid2

  The `!` indicates that the request originated on
  that node.
* To propigate a request, set -1 * (TTL+1) in the line
  for the uuid of the repository that is propigating it.
  This should be done as part of the git-annex branch merge,
  so if a location tracking file is merged, any open requests
  get propigated to the current repository automatically.
* When a requested file reaches a node that requested it,
  the location is set to 1; this automatically clears the
  request.
* When a file has no more originating requests, clear all
  the copied requests:

	time 1 uuid1
	time -2 uuid2

  Becomes:

	time 1 uuid1
	time' 0 uuid2

## generating requests

	git annex request [file...]

Indicates that the file is wanted in the current repository.

(git annex get could also do this on failure, or suggest doing this)

## acting on requests

Add a preferred content expression that looks at request data:

	requestedby=N

Matches files that have been requested by at least N nodes.

	requested

Matches files that the current node has requested.

### Example preferred content expressions

For an immobile node that accumulates files it requests, and also
temporarily stores files requested by other such nodes:

	present or requestedby=1

For a node that only transfers files between the immobile nodes:

	requestedby=1

For an immobile node that only accumulates files it requests, but never
stores files requested by other nodes:

	present or requested

TODO: Would be nice to be able to prioritize files that more nodes are
requesting, or that have some urgent flag set. But currently there is no
way to do that; content is either preferred or not preferred.


code:
[[!comment format=mdwn
 username="http://johan.kiviniemi.name/"
 nickname="Johan"
 subject="comment 1"
 date="2014-05-07T06:27:10Z"
 content="""
“Nodes generate a request for a specific file they want”

It would be also convenient to be able to “push” a file to another node you do not have direct access to by creating a request for that file on behalf of the node.
"""]]


code:
-- Simulation of non-flood syncing of content, across a network of nodes.

module Main where

import System.Random
import Control.Monad.Random
import Control.Monad
import Control.Applicative
import Data.Ratio
import Data.Ord
import Data.List
import Data.Maybe
import qualified Data.Set as S
import qualified Data.Map.Strict as M

{-
 - Tunable values
 -}

totalFiles :: Int
totalFiles = 100

-- How likely is a given file to be wanted by any particular node?
probabilityFilesWanted :: Probability
probabilityFilesWanted = 0.10

-- How many different locations can each transfer node move between?
-- (Min, Max)
transferDestinationsRange :: (Int, Int)
transferDestinationsRange = (2, 3)

-- Controls how likely transfer nodes are to move around in a given step
-- of the simulation.
-- (They actually move slightly less because they may start to move and
-- pick the same location they are at.)
-- (Min, Max)
transferMoveFrequencyRange :: (Probability, Probability)
transferMoveFrequencyRange = (0.10, 1.00)

-- counts both immobile and transfer nodes as hops, so double Vince's
-- theoretical TTL of 3.
-- (30% loss on mocambos network w/o ttl of 4!)
maxTTL :: TTL
maxTTL = TTL (4 * 2)

numImmobileNodes :: Int
numImmobileNodes = 10

numTransferNodes :: Int
numTransferNodes = 20

numSteps :: Int
numSteps = 100

-- IO code
main :: IO ()
main = do
--	initialnetwork <- evalRandIO (seedFiles totalFiles =<< genNetwork)
	initialnetwork <- evalRandIO (seedFiles totalFiles =<< mocambosNetwork)
	networks <- evalRandIO (simulate numSteps initialnetwork)
	let finalnetwork = last networks
	putStrLn $ summarize initialnetwork finalnetwork
	putStrLn "location history of file 1:"
	print $ trace (traceHaveFile (File 1)) networks
	putStrLn "request history of file 1:"
	print $ trace (traceWantFile (File 1)) networks
-- Only pure code below :)

data Network = Network (M.Map NodeName ImmobileNode) [TransferNode]
	deriving (Show, Eq)

data ImmobileNode = ImmobileNode NodeRepo
	deriving (Show, Eq)

type NodeName = String

type Route = [NodeName]

data TransferNode = TransferNode
	{ currentlocation :: NodeName
	, possiblelocations :: [NodeName]
	, movefrequency :: Probability
	, transferrepo :: NodeRepo
	}
	deriving (Show, Eq)

data NodeRepo = NodeRepo
	{ wantFiles :: [Request]
	, haveFiles :: S.Set File
	, satisfiedRequests :: S.Set Request
	}
	deriving (Show, Eq)

data File = File Int
	deriving (Show, Eq, Ord)

randomFile :: (RandomGen g) => Rand g File
randomFile = File <$> getRandomR (0, totalFiles)

data Request = Request File TTL
	deriving (Show, Ord)

-- compare ignoring TTL
instance Eq Request where
	(Request f1 _) == (Request f2 _) = f1 == f2

requestedFile :: Request -> File
requestedFile (Request f _) = f

requestTTL :: Request -> TTL
requestTTL (Request _ ttl) = ttl

data TTL = TTL Int
	deriving (Show, Eq, Ord)

incTTL :: TTL -> TTL
incTTL (TTL t) = TTL (t + 1)

decTTL :: TTL -> TTL
decTTL (TTL t) = TTL (t - 1)

staleTTL :: TTL -> Bool
staleTTL (TTL t) = t < 1

-- Origin of a request starts one higher than max, since the TTL
-- will decrement the first time the Request is transferred to another node.
originTTL :: TTL
originTTL = incTTL maxTTL

randomRequest :: (RandomGen g) => Rand g Request
randomRequest = Request
	<$> randomFile
	<*> pure originTTL

type Probability = Float

randomProbability :: (RandomGen g) => Rand g Probability
randomProbability = getRandomR (0, 1)

-- Returns the state of the network at each step of the simulation.
simulate :: (RandomGen g) => Int -> Network -> Rand g [Network]
simulate n net = go n [net]
  where
	go 0 nets = return (reverse nets)
	go c (prev:nets) = do
		new <- step prev
		go (c - 1) (new:prev:nets)

-- Each step of the simulation, check if each TransferNode wants to move,
-- and if so:
--   1. It and its current location exchange their Requests.
--   2. And they exchange any requested files.
--   3. Move it to a new random location.
--
-- Note: This implementation does not exchange requests between two
-- TransferNodes that both arrive at the same location at the same step,
-- and then move away in the next step.
step :: (RandomGen g) => Network -> Rand g Network
step (Network immobiles transfers) = go immobiles [] transfers
  where
	go is c [] = return (Network is c)
	go is c (t:ts) = do
		r <- randomProbability
		if movefrequency t <= r
			then case M.lookup (currentlocation t) is of
				Nothing -> go is (c ++ [t]) ts
				Just currentloc -> do
					let (currentloc', t') = merge currentloc t
					t'' <- move t'
					go (M.insert (currentlocation t) currentloc' is) (c ++ [t'']) ts
			else go is (c ++ [t]) ts

merge :: ImmobileNode -> TransferNode -> (ImmobileNode, TransferNode)
merge (ImmobileNode ir) t@(TransferNode { transferrepo = tr }) =
	( ImmobileNode (go ir tr)
	, t { transferrepo = go tr ir }
	)
  where
	go r1 r2 = r1
		{ wantFiles = wantFiles'
		, haveFiles = haveFiles'
		, satisfiedRequests = satisfiedRequests' `S.union` checkSatisfied wantFiles' haveFiles'
		}
	  where
	  	wantFiles' = foldr addRequest (wantFiles r1) (wantFiles r2)
		haveFiles' = S.foldr (addFile wantFiles' satisfiedRequests') (haveFiles r1) (haveFiles r2)
		satisfiedRequests' = satisfiedRequests r1 `S.union` satisfiedRequests r2

-- Adds a file to the set, when there's a request for it, and the request
-- has not already been satisfied.
addFile :: [Request] -> S.Set Request -> File -> S.Set File -> S.Set File
addFile rs srs f fs
	| any (\sr -> f == requestedFile sr) (S.toList srs) = fs
	| any (\r -> f == requestedFile r) rs = S.insert f fs
	| otherwise = fs

-- Checks if any requests have been satisfied, and returns them,
-- to be added to satisfidRequests
checkSatisfied :: [Request] -> S.Set File -> S.Set Request
checkSatisfied want have = S.fromList (filter satisfied want)
  where
	satisfied r = requestTTL r == originTTL && S.member (requestedFile r) have

-- Decrements TTL, and avoids adding request with a stale TTL, or a
-- request for an already added file with the same or a lower TTL.
addRequest :: Request -> [Request] -> [Request]
addRequest (Request f ttl) rs
	| staleTTL ttl' = rs
	| any (\r -> requestTTL r >= ttl) similar = rs
	| otherwise = r' : other
  where
	ttl' = decTTL ttl
	r' = Request f ttl'
	(other, similar) = partition (/= r') rs

move :: (RandomGen g) => TransferNode -> Rand g TransferNode
move t = do
	newloc <- randomfrom (possiblelocations t)
	return $ t { currentlocation = newloc }

genNetwork :: (RandomGen g) => Rand g Network
genNetwork = do
	let immobiles = M.fromList (zip (map show [1..]) (replicate numImmobileNodes emptyImmobile))
	transfers <- sequence (replicate numTransferNodes (mkTransfer $ M.keys immobiles))
	return $ Network immobiles transfers

emptyImmobile :: ImmobileNode
emptyImmobile = ImmobileNode (NodeRepo [] S.empty S.empty)

mkTransfer :: (RandomGen g) => [NodeName] -> Rand g TransferNode
mkTransfer immobiles = do
  	-- Transfer nodes are given random routes. May be simplistic.
	-- Also, some immobile nodes will not be serviced by any transfer nodes.
	numpossiblelocs <- getRandomR transferDestinationsRange
	possiblelocs <- sequence (replicate numpossiblelocs (randomfrom immobiles))
	mkTransferBetween possiblelocs

mkTransferBetween :: (RandomGen g) => [NodeName] -> Rand g TransferNode
mkTransferBetween possiblelocs = do
	currentloc <- randomfrom possiblelocs
	movefreq <- getRandomR transferMoveFrequencyRange
	-- transfer nodes start out with no files or requests in their repo
	let repo = (NodeRepo [] S.empty S.empty)
	return $ TransferNode currentloc possiblelocs movefreq repo

randomfrom :: (RandomGen g) => [a] -> Rand g a
randomfrom l = do
	i <- getRandomR (1, length l)
	return $ l !! (i - 1)

-- Seeds the network with the given number of files. Each file is added to
-- one of the immobile nodes of the network at random. And, one other node,
-- at random, is selected which wants to get the file.
seedFiles :: (RandomGen g) => Int -> Network -> Rand g Network
seedFiles 0 network = return network
seedFiles n network@(Network m t) = do
	(origink, ImmobileNode originr) <- randnode
	(destinationk, ImmobileNode destinationr) <- randnode
	let file = File n
	let origin = ImmobileNode $ originr
		{ haveFiles = S.insert file (haveFiles originr) }
	let destination = ImmobileNode $ destinationr
		{ wantFiles = Request file originTTL : wantFiles destinationr }
	let m' = M.insert origink origin $
		M.insert destinationk destination m
	seedFiles (n - 1) (Network m' t)
  where
	randnode = do
		k <- randomfrom (M.keys m)
		return (k, fromJust $ M.lookup k m)

summarize :: Network -> Network -> String
summarize _initial@(Network origis _) _final@(Network is _ts) = format
	[ ("Total wanted files",
		show (sum (overis (length . findoriginreqs . wantFiles . repo))))
	, ("Wanted files that were not transferred to requesting node",
		show (sum (overis (S.size . findunsatisfied . repo))))
	, ("Nodes that failed to get files",
		show (map withinitiallocs $ filter (not . S.null . snd)
			(M.toList $ M.map (findunsatisfied . repo) is)))
	, ("Total number of files on immobile nodes at end",
		show (overis (S.size . haveFiles . repo)))
	--, ("Immobile nodes at end", show is)
	]
  where
  	findoriginreqs = filter (\r -> requestTTL r == originTTL)
	findunsatisfied r =
		let wantedfs = S.fromList $ map requestedFile (findoriginreqs (wantFiles r))
		in S.difference wantedfs (haveFiles r)
	repo (ImmobileNode r) = r
	overis f = map f $ M.elems is
	format = unlines . map (\(d, s) -> d ++ ": " ++ s)

	withinitiallocs (name, missingfiles) = (name, S.map addinitialloc missingfiles)
	addinitialloc f = (f, M.lookup f initiallocs)

	initiallocs = M.fromList $
		concatMap (\(k, v) -> map (\f -> (f, k)) (S.toList $ haveFiles $ repo v)) $
			M.toList origis

trace :: (Network -> S.Set NodeName) -> [Network] -> String
trace tracer networks = show $ go [] S.empty $ map tracer networks
  where
	go c old [] = reverse c
	go c old (new:l) = go ((S.toList $ new `S.difference` old):c) new l

traceHaveFile :: File -> Network -> S.Set NodeName
traceHaveFile f (Network m _) = S.fromList $ M.keys $
	M.filter (\(ImmobileNode r) -> f `S.member` haveFiles r) m

traceWantFile :: File -> Network -> S.Set NodeName
traceWantFile f (Network m _) = S.fromList $ M.keys $
	M.filter (\(ImmobileNode r) -> any wantf (wantFiles r)) m
  where
	wantf (Request rf _ttl) = rf == f

mocambosNetwork :: (RandomGen g) => Rand g Network
mocambosNetwork = do
	let major = map (immobilenamed . fst) communities
	let minor = map immobilenamed (concatMap snd communities)
	majortransfer <- mapM mkTransferBetween majorroutes
	minortransfer <- mapM mkTransferBetween (concatMap minorroutes (concat (replicate 5 communities)))
	return $ Network
		(M.fromList (major++minor))
		(majortransfer ++ minortransfer)
  where
	immobilenamed name = (name, emptyImmobile)

	-- As a simplification, this only makes 2 hop routes, between minor
	-- and major communities; no 3-legged routes.
	minorroutes :: (NodeName, [NodeName]) -> [Route]
	minorroutes (major, minors) = map (\n -> [major, n]) minors

communities :: [(NodeName, [NodeName])]
communities =
	[ ("Tainá/SP",
		[ "badtas"
		, "vauedo ribera"
		, "cofundo"
		, "jao"
		, "fazenda"
		]
	  )
	, ("Odomode/RS",
		[ "moradadapaz"
		, "pelotas"
		]
	  )
	, ("MercadoSul/DF",
		[ "mesquito"
		, "kalungos"
		]
	  )
	, ("Coco/PE",
		[ "xambá"
		, "alafin"
		, "terreiaos"
		]
	  )
	, ("Linharinho/ES",
		[ "monte alegne"
		]
	  )
	, ("Boneco/BA",
		[ "barroso"
		, "lagoa santa"
		, "terravista"
		]
	  )
	, ("Zumbidospalmanes/NA",
		[ "allantana"
		]
	  )
	, ("Casa Pneta/PA",
		[ "marajó"
		]
	  )
	, ("Purarue/PA",
		[ "oriaminá"
		]
	  )
	, ("Madiba/NET", [])
	]

majorroutes :: [Route]
majorroutes =
	-- person's routes
	[ ["Tainá/SP", "Odomode/RS"]
	, ["Tainá/SP", "MercadoSul/DF"]
	, ["MercadoSul/DF", "Boneco/BA"]
	, ["MercadoSul/DF", "Zumbidospalmanes/NA"]
	, ["Zumbidospalmanes/NA", "Casa Pneta/PA"]
	, ["Casa Pneta/PA", "Purarue/PA"]
	, ["Casa Pneta/PA", "Linharinho/ES"]
	, ["Boneco/BA", "Coco/PE"]
	-- internet connections
	, ["Tainá/SP", "MercadoSul/DF", "Coco/PE", "Purarue/PA", "Odomode/RS", "Madiba/NET"]
	, ["Tainá/SP", "MercadoSul/DF", "Coco/PE", "Purarue/PA", "Odomode/RS", "Madiba/NET"]
	, ["Tainá/SP", "MercadoSul/DF", "Coco/PE", "Purarue/PA", "Odomode/RS", "Madiba/NET"]
	, ["Tainá/SP", "MercadoSul/DF", "Coco/PE", "Purarue/PA", "Odomode/RS", "Madiba/NET"]
	, ["Tainá/SP", "MercadoSul/DF", "Coco/PE", "Purarue/PA", "Odomode/RS", "Madiba/NET"]
	]


code:
## roadmap

Now in the
[sustaining git-annex development](https://campaign.joeyh.name/) year
(starting September 2013).

* Month 1 [[!traillink assistant/encrypted_git_remotes]]
* Month 2 [[!traillink assistant/disaster_recovery]]
* Month 3 [[!traillink direct_mode]] guard [[!traillink assistant/upgrading]]
* Month 4 [[!traillink assistant/windows text="Windows webapp"]], Linux arm, external special remotes
* Month 5 user-driven features and polishing
* Month 6 get Windows out of beta, [[!traillink design/metadata text="metadata and views"]]
* Month 7 user-driven features and polishing
* Month 8 [[!traillink git-remote-daemon]]
* Month 9 Brazil!, [[!traillink assistant/sshpassword]]
* Month 10 polish [[assistant/Windows]] port
* Month 11 [[!traillink assistant/chunks]]
* **Month 12** user-driven features and polishing

Deferred until later:

* Month XX [[!traillink assistant/deltas]], [[!traillink assistant/gpgkeys]]
* Month XX [[!traillink assistant/telehash]]


code:
Work on git-annex is [crowdfunded](https://campaign.joeyh.name/).
Joey blogs about his progress here on a semi-daily basis.

[[!sidebar content="""
[[!calendar type="month" pages="page(devblog/*)"]]
[[!calendar type="month" month="-1" pages="page(devblog/*)"]]
[[!calendar type="month" month="-2" pages="page(devblog/*)"]]
"""]]

[[!inline pages="page(devblog/*)" show=0]]


code:
Implemented `git annex forget --drop-dead`, which is finally a way to
remove all references to old repositories that you've marked as dead.

I've still not merged in the `forget` branch, because I developed this
while slightly ill, and have not tested it very well yet.


code:
John Millikin came through and fixed that haskell-gnutls segfault
on OSX that I developed a reproducible test case for the other day.
It's a bit hard to test, since the bug doesn't always happen, but the
fix is already deployed for Mountain Lion autobuilder.

However, I then found another way to make haskell-gnutls segfault, more
reliably on OSX, and even sometimes on Linux. Just entering the wrong XMPP
password in the assistant can trigger this crash. Hopefully John will work
his magic again.

---

Meanwhile, I fixed the sync-after-forget problem. Now sync always forces
its push of the git-annex branch (as does the assistant). I considered but
rejected having sync do the kind of uuid-tagged branch push that the
assistant sometimes falls back to if it's failing to do a normal sync. It's
ugly, but worse, it wouldn't work in the workflow where multiple clients
are syncing to a central bare repository, because they'd not pull down the
hidden uuid-tagged branches, and without the assistant running on the
repository, nothing would ever merge their data into the git-annex branch.
Forcing the push of synced/git-annex was easy, once I satisfied myself
that it was always ok to do so.

Also factored out a module that knows about all the different log files
stored on the git-annex branch, which is all the support infrastructure
that will be needed to make `git annex forget --drop-dead` work. Since this
is basically a routing module, perhaps I'll get around to making it use
a nice bidirectional routing library like
[Zwaluw](http://hackage.haskell.org/package/Zwaluw) one day.


code:
Yesterday I spent making a release, and shopping for a new laptop, since
this one is dying. (Soon I'll be able to compile git-annex fast-ish! Yay!)
And thinking about [[todo/wishlist:_dropping_git-annex_history]].

Today, I added the `git annex forget` command. It's currently been lightly
tested, seems to work, and is living in the `forget` branch until I gain
confidence with it. It should be perfectly safe to use, even if it's buggy,
because you can use `git reflog git-annex` to pull out and revert to an old
version of your git-annex branch. So if you're been wanting this feature,
please beta test!

----

I actually implemented something more generic than just forgetting git
history. There's now a whole mechanism for git-annex doing distributed
transitions of whatever sort is needed.

There were several subtleties involved in distributed transitions:

First is how to tell when a given transition has already been done on a
branch. At first I was thinking that the transition log should include the
sha of the first commit on the old branch that got rewritten. However, that
would mean that after a single transition had been done, every git-annex
branch merge would need to look up the first commit of the current branch,
to see if it's done the transition yet. That's slow! Instead, transitions
are logged with a timestamp, and as long as a branch contains a transition
with the same timestamp, it's been done.

A really tricky problem is what to do if the local repository has
transitioned, but a remote has not, and changes keep being made to the
remote. What it does so far is incorporate the changes from the remote into
the index, and re-run the transition code over the whole thing to yeild a
single new commit. This might not be very efficient (once I write the more
full-featured transition code), but it lets the local repo keep up with
what's going on in the remote, without directly merging with it (which
would revert the transition). And once the remote repository has its
git-annex upgraded to one that knows about transitions, it will finish up
the transition on its side automatically, and the two branches will once
again merge.

Related to the previous problem, we don't want to keep trying to merge
from a remote branch when it's not yet transitioned. So a blacklist is
used, of untransitioned commits that have already been integrated.

One really subtle thing is that when the user does a transition more
complicated than `git annex forget`, like the `git annex forget --dead`
that I need to implement to forget dead remotes, they're not just telling
git-annex to forget whatever dead remotes it knows right now. They're
actually telling git-annex to perform the transition one time on every
existing clone of the repository, at some point in the future. Repositories
with unfinished transitions could hang around for years, and at some future
point when git-annex runs in the repository again, it would merge in the
current state of the world, and re-do the transition. So you might tell it
to forget dead remotes today, and then the very repository you ran that in
later becomes dead, and a long-slumbering repo wakes up and forgets about
the repo that started the whole process! I hope users don't find this
massively confusing, but that's how the implementation works right now.

----

I think I have at least two more days of work to do to finish up this
feature.

* I still need to add some extra features like forgetting about dead remotes,
  and forgetting about keys that are no longer present on any remote.

* After `git annex forget`, `git annex sync`
  will fail to push the synced/annex branch to remotes, since the branch
  is no longer a fast-forward of the old one. I will probably fix this by
  making `git annex sync` do a fallback push of a unique branch in this case,
  like the assistant already does. Although I may need to adjust that code
  to handle this case, too..

* For some reason the automatic transitioning code triggers
  a "(recovery from race)" commit. This is certainly a bug somewhere,
  because you can't have a race with only 1 participant.

----

Today's work was sponsored by Richard Hartmann.


code:
[[!comment format=mdwn
 username="Sandra.Devil"
 ip="77.172.73.184"
 subject="New laptop"
 date="2013-09-01T09:38:32Z"
 content="""
What is the new laptop you are going to use? Specs please :)
"""]]


code:
[[!comment format=mdwn
 username="stp"
 ip="188.193.207.34"
 subject="Update one forgetting keys no longer present"
 date="2014-02-17T23:21:49Z"
 content="""
I have some repos where due to some hiccups file versions (not in the working tree anymore) were lost and now they come up again and again when fsck is running.
So I would be happy if I could make my repos forget these not available files via \"git annex forget $key\" and perhaps even have a better solution to show all objects with numcopies=0.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.172"
 subject="comment 3"
 date="2014-02-20T18:54:48Z"
 content="""
@stp, It seems to me if you just delete the symlinks in your git repository that point to the lost files, `git annex fsck` will shut up.
"""]]


code:
[[!comment format=mdwn
 username="stp"
 ip="84.56.21.11"
 subject="comment 4"
 date="2014-02-20T21:07:41Z"
 content="""
Yeah true if I remove symlinks from the history (as I understand your suggestion) it would work. I just wanted to suggest that it could be something useful for the git annex forget function as it already cleans out old dead repos and other things.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.172"
 subject="comment 5"
 date="2014-02-20T21:14:21Z"
 content="""
You don't need to delete them from the history, just from the branch you're running `git annex fsck` in.
"""]]


code:
[[!comment format=mdwn
 username="stp"
 ip="84.56.21.11"
 subject="comment 6"
 date="2014-02-20T22:19:12Z"
 content="""
As discussed on irc.
Fsck --all does check more then the working tree and therefore for fsck to not complain this would be a worthy feature to be added. (git annex forget $key)
"""]]


code:
[[!comment format=mdwn
 username="stp"
 ip="84.56.21.11"
 subject="New findings"
 date="2014-02-24T12:28:03Z"
 content="""
Another thing I found, which was annoying is that I have objects in my annex not tracked anywhere it seems.
\"git annex fsck --all\" complains about not having access to the object. \"git log --stat -S '$key'\" doesn't have any record. \"git annex fsck\" has no issues and \"git annex unused\" comes up empty too.
I'm not sure where these objects still reside or why how to remove this annoying failure.

So not only should \"git annex forget $key\" remove references from within all branches, but should also clean up the aforementioned loose objects, which are neither unused, nor available, nor referenced.
"""]]


code:
[[!comment format=mdwn
 username="stp"
 ip="188.193.207.34"
 subject="Any update on cleaning up commands?"
 date="2014-03-21T16:32:43Z"
 content="""
Is there any update on cleaning up object/file references to objects/content not at all present and lost. I would love my git annex fsck --all to show current failures and not these old files all the time.
Thanks
"""]]


code:
Spent the day building this new feature, which makes `git annex sync --content`
do the same synchronization of file contents (to satisfy preferred content
settings) that the assistant does. The result has not been tested a lot
yet, but seems to work well.


code:
In order to remove some hackishness in `git annex sync --content`, I
finally fixed a bad design decision I made back at the very beginning
(before I really knew haskell) when I built the command seek code, which
had led to a kind of inversion of control. This took most of a night, but
it made a lot of code in git-annex clearer, and it makes the command
seeking code much more flexible in what it can do. Some of the oldest, and
worst code in git-annex was removed in the process.

Also, I've been reworking the numcopies configuration, to allow for a
[[todo/preferred_content_numcopies_check]]. That will let the assistant,
as well as `git annex sync --content` proactively make copies when
needed in order to satisfy numcopies.

As part of this, `git config annex.numcopies` is deprecated, and there's a
new `git annex numcopies N` command that sets the numcopies value that will
be used by any clone of a repository.

I got the preferred content checking of numcopies working too. However,
I am unsure if checking for per-file .gitattributes annex.numcopies
settings will make preferred content expressions be, so I have left
that out for now.

Today's work was sponsored by Josh Taylor.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 1"
 date="2014-01-21T00:20:34Z"
 content="""
Some more details on

* when the old mechanism will stop working
* how you will ensure interoperability in repo sets which have different versions installed in different places
* which option weighs more heavily

would be appreciated.

Richard
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.249"
 subject="comment 2"
 date="2014-01-21T02:23:00Z"
 content="""
I don't plan to remove .git/config annex.numcopies support. It is 5 lines of code.

A fundamental problem with the .git/config annex.numcopies setting is that it can be inconsistently set between repositories. Therefore, there is no worse potential with interoperability if you use differing versions of git-annex that support and do not support the new global numcopies setting. If this was a concern though, you can use .gitattributes to set annex.numcopies for *, which all recent versions of git-annex support.

The old git/config annex.numcopies setting is overridden as soon as you use `git annex numcopies`. In turn `git annex numcopies` is overridden by any relevant .gitattributes annex.numcopies setting. This is documented on the man page.
"""]]


code:
Worked on cleaning up and reorganizing all the code that handles numcopies
settings. Much nicer now. Fixed some bugs.

As expected, making the preferred content numcopies check look at
.gitattributes slows it down significantly. So, exposed both the slow and
accurate check and a faster version that ignores .gitattributes.

Also worked on the test suite, removing dependencies between tests.
This will let tasty-rerun be used later to run only previously failing
tests.


code:
A big missing peice of the assistant is doing something about the content
of old versions of files, and deleted files. In direct mode, editing or
deleting a file necessarily loses its content from the local repository,
but the content can still hang around in other repositories. So, the
assistant needs to do something about that to avoid eating up disk space
unnecessarily.

I built on recent work, that lets preferred content expressions be matched
against keys with no associated file. This means that I can run unused keys
through all the machinery in the assistant that handles file transfers, and
they'll end being moved to whatever repository wants them. To control which
repositories do want to retain unused files, and which not, I added a
`unused` keyword to preferred content expressions. Client repositories and
transfer repositories do not want to retain unused files, but backup etc
repos do.

One nice thing about this `unused` preferred content implementation is that
it doesn't slow down normal matching of preferred content expressions at
all. Can you guess why not? See [[!commit 4b55afe9e92c045d72b78747021e15e8dfc16416]]

So, the assistant will run `git annex unused` on a daily basis, and
cause unused files to flow to repositories that want them. But what if no
repositories do? To guard against filling up the local disk, there's
a `annex.expireunused` configuration setting, that can cause old unused
files to be deleted by the assistant after a number of days.

I made the assistant check if there seem to be a lot of unused files piling
up. (1000+, or 10% of disk used by them, or more space taken by unused files
than is free.) If so, it'll pop up an alert to nudge the user to configure
annex.expireunused.

Still need to build the UI to configure that, and test all of this.

Today's work was sponsored by Samuel Tardieu.


code:
Built the UI to manage unused files.

[[!img assistant/unused.png]]

Testing yesterday's work, I found several problems that prevented the
assistant from moving unused files around, and fixed them. It seems to be
working pretty well now.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm7AuSfii_tCkLyspL6Mr0ATlO6OxLNYOo"
 nickname="Georg"
 subject="state of windows"
 date="2014-01-28T10:29:41Z"
 content="""
Any idea when this stuff will end in the windows build? I am currently fiddling around with git-annex on windows (btw: great work on that so far!) and the unused files config would be highly valueable to me, since I want to mostly use it to push files out to my server from my work machine or my windows tablet, while preventing to clog up too much space. And since I am lazy, I would love to leave the house keeping to git-annex ;)

An additional question on the windows build: which of the backends can actually make use of windows proxy settings? I guess if webdav was in the list of supported backends, it would work fine, since it is only http, but since it is missing, which of the others would work through just a standard configured squid proxy? For my work machine this is the only way to reach the outside world, doing https to my web server.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.199"
 subject="comment 2"
 date="2014-01-28T20:58:32Z"
 content="""
AFAICS, the Windows build does include WebDAV support.

I don't know about Windows proxy stuff. If it requires the program to do something, git-annex almost certianly does not do it.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm7AuSfii_tCkLyspL6Mr0ATlO6OxLNYOo"
 nickname="Georg"
 subject="comment 3"
 date="2014-01-29T08:22:13Z"
 content="""
well, at least in the assistant there is no selection to add a webdav repository - only box.com, rsync.net, S3, remote ssh, and the archives on glazier or internet archive. It is the binary version from 16th this month. I thought it was still excluded due to the compiler crashes you wrote about, because it doesn't show up in the webapp as option. I was scratching my head anyway about that windows build, because there was nothing about setting up encryption in the configuration in the webapp.

Proxy support on Windows is kinda like proxy support on linux when you pull the proxy from gnome settings - just it is pulled from the registry. I don't know how Haskell HTTP client libraries handle it, only know it for Python - there the client libraries usually grab proxy settings automatically from the system configuration.

Another option which could work for me to get through our firewall at work would be ssh - I can use ssh with proxy commands to reach the outside, but didn't yet find the used ssh configs on the windows git-annex installation - it doesn't use the one in ~/.ssh, I added my proxy commands there, but that isn't used.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm7AuSfii_tCkLyspL6Mr0ATlO6OxLNYOo"
 nickname="Georg"
 subject="comment 4"
 date="2014-01-29T08:26:19Z"
 content="""
Oh, just noticed in the console window of the assistant:

ssh-keygen: /home/xgwsbae/.ssh/known_hosts: No such file or directory
ssh-keygen: /home/xgwsbae/.ssh/known_hosts: No such file or directory

Home directory is C:/Users/xgwsbae/ (git bash gives /c/Users/xgwsbae in $HOME) ... looks like something got confuzzled between Windows and Unix there.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.199"
 subject="comment 5"
 date="2014-01-29T17:58:26Z"
 content="""
Some things (not git-annex) default to /home/$USER when appropriate env vars are not set. There is a bug open about that, [[bugs/assistant_using_the_incorrect_path_on_windows?]], although I don't know what it makes sense for git-annex to do to deal with it.

box.com *is* a webdav repository. It is the only webdav repository the webapp currently supports setting up, although you can configure any webdav remote you like at the command line and the assistant will then be able to use it.

Will be curious to hear if the http-conduit library used for webdav supports proxies. If not, that would certianly seem a good place to file a bug. Unofrtunately, git-annex uses 2 or 3 other http stacks in other places too.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm7AuSfii_tCkLyspL6Mr0ATlO6OxLNYOo"
 nickname="Georg"
 subject="comment 6"
 date="2014-01-30T10:52:09Z"
 content="""
Well, according to the hackage documentation of http conduit it seems it at least supports proxies when youre code gives it the proxy host and port.

But the link about the problem with environment settings helped me to get around my immediate problem by just running annex from the git bash (which is prefereably to me most of the time anyway). So I could set up my annex on my work machine and just tunnel out with ProxyCommand via our ssh gateway. Yay!
"""]]


code:
With yesterday's release, I'm pretty much done with the month's work. Since
there was no particular goal this month, it's been a grab bag of features
and bugfixes. Quite a lot of them in this last release.

I'll be away the next couple of days.. But got a start today on the next
part of the roadmap, which is planned to be all about Windows and Android
porting. Today, it was all about lock files, mostly on Windows.

Lock files on Windows are horrific. I especially like that programs that
want to open a file, for any reason, are encouraged in the official
documentation to retry repeatedly if it fails, because some other
random program, like a virus checker, might have opened the file first.

Turns out Windows does support a shared file read mode. This was
just barely enough for me to implement both shared and exclusive
file locking a-la-flock.

Couldn't avoid a busy wait in a few places that block on a lock.
Luckily, these are few, and the chances the lock will be taken for a long
time is small. (I did think about trying to watch the file for close events
and detect when the lock was released that way, but it seemed much too
complicated and hard to avoid races.)

Also, Windows only seems to support mandatory locks, while all locking in
git-annex needs to be advisory locks. Ie, git-annex's locking shouldn't
prevent a program from opening an annexed file! To work around that,
I am using dedicated lock files on Windows.

Also switched direct mode's annexed object locking to use dedicated lock
files. AFAICS, this was pretty well broken in direct mode before.


code:
While I've not been blogging over what amounted to a long weekend, looking
over the changelog, there were quite a few things done. Mostly various
improvements and fixes to `git annex sync --content`.

Today, got the test suite to pass on Windows 100% again.


code:
A more test driven day than usual. Yesterday I noticed a test case was
failing on Windows in a way not related to what it was intended to test,
and fixed the test case to not fail.. But knew I'd need to get to the
bottom of what broke it eventually.

Digging into that today, I eventually (after rather a long time stuck)
determined the bug involved automatic conflict resolution, but only
happened on systems without symlink support. This let me reproduce it on
FAT outside Windows and do some fast TDD iterations in a much less
unwieldly environment and fix the bug.


code:
Added a new feature that started out with me wanting a way to undo a
`git-annex drop`, but turned into something rather more powerful. The `--in`
option can now be told to match files that were in a repository at some
point in the past. For example, `git annex get --in=here@{yesterday}` will
get any files that have been dropped over the past day.

While git-annex's location tracking info is stored in git and thus versioned,
very little of it makes use of past versions of the location tracking info
(only `git annex log`). I'm happy to have finally found a use for it!

OB Windows porting: Fixed a bug in the symlink calculation code.
Sounds simple; took 2 hours!

Also various bug triage; updated git version on OSX; forwarded bug about
DAV-0.6 being broken upstream; fixed a bug with initremote in
encryption=pubkey mode. Backlog is 65 messages.

---

Today's work was sponsored by Brock Spratlen.


code:
git-annex has been using MissingH's `absNormPath` forever, but that's
not very maintained and doesn't work on Windows. I've been
wanting to get rid of it for some time, and finally did today, writing a
`simplifyPath` that does the things git-annex needs and will work with all
the Windows filename craziness, and takes advantage of the more modern
System.FilePath to be quite a simple peice of code. A QuickCheck test found
no important divergences from absNormPath. A good first step to making
git-annex not depend on MissingH at all.

That fixed one last Windows bug that was disabled in the test suite:
`git annex add ..\subdir\file` will now work.

I am re-installing the Android autobuilder for 2 reasons: I noticed I had
accidentially lost a patch to make a library use the Android SSL cert directory,
and also a new version of GHC is very near to release and so it makes sense
to update.

Down to 38 messages in the backlog.


code:
Fixed a typo that broke automatic youtube video support in `addurl`.

----

Now there's an easy way to get an overview of how close your repository
is to meeting the configured numcopies settings (or when it exceeds them).

<pre>
# time git annex status .
[...]
numcopies stats:
	numcopies +0: 6686
	numcopies +1: 3793
	numcopies +3: 3156
	numcopies +2: 2743
	numcopies -1: 1242
	numcopies -4: 1098
	numcopies -3: 1009
	numcopies +4: 372
</pre>

This does make `git annex status` slow when run on a large directory tree,
so --fast disables that.


code:
Last night I tracked down and fixed a bug in the DAV library that has been
affecting WebDAV remotes. I've been deploying the fix for that today,
including to the android and arm autobuilders. While I finished a clean
reinstall of the android autobuilder, I ran into problems getting a clean
reinstall of the arm autobuilder (some type mismatch error building
yesod-core), so manually fixed its DAV for now.

The WebDAV fix and other recent fixes makes me want to make a release soon,
probably Monday.

ObWindows: Fixed git-annex to not crash when run on Windows
in a git repository that has a remote with a unix-style path
like "/foo/bar". Seems that not everything aggrees on whether such a path
is absolute; even sometimes different parts of the same library disagree!

[[!format haskell """
import System.FilePath.Windows

prop_windows_is_sane :: Bool
prop_windows_is_sane = isAbsolute upath || ("C:\\STUFF" </> upath /= upath)
  where upath = "/foo/bar"
"""]]

Perhaps more interestingly, I've been helping dxtrish port git-annex to
OpenBSD and it seems most of the way there.


code:
Pushed out the new release. This is the first one where I consider the
git-annex command line beta quality on Windows.

Did some testing of the webapp on Windows, trying out every part of the UI.
I now have eleven todo items involving the webapp listed in
[[todo/windows_support]]. Most of them don't look too bad to fix.


code:
There's a new design document for letting git-annex store arbitrary
metadata. The really neat thing about this is the user can check out only
files matching the tags or values they care about, and get an automatically
structuted file tree layout that can be dynamically filtered. It's going to
be awesome! [[design/metadata]]

In the meantime, spent most of today working on Windows. Very good
progress, possibly motivated by wanting to get it over with so I can spend
some time this month on the above. ;)

* webapp can make box.com and S3 remotes. This just involved fixing a hack
  where the webapp set environment variables to communicate creds to
  initremote. Can't change environment on Windows (or I don't know how to).
* webapp can make repos on removable drives.
* `git annex assistant --stop` works, although this is not likely to really
  be useful
* The source tree now has 0 `func = error "Windows TODO"` type stubbed out
  functions to trip over.


code:
Built the core data types, and log for metadata storage. Making metadata
union merge well is tricky, but I have a design I'm happy with, that will
allow distributed changes to metadata.

Finished up the day with a `git annex metadata` command to get/set metadata
for a file.

This is all the goundwork needed to begin experimenting with generating
git branches that display different metadata-driven views of annexed files.


code:
Windows porting all day. Fixed a lot of issues with the webapp,
so quite productive. Except for the 2 hours wasted finding a way to kill a
process by PID from Haskell on Windows.

Last night, made `git annex metadata` able to set metadata on a whole
directory or list of files if desired. And added a `--metadata field=value`
switch (and corresponding preferred content terminal) which limits
git-annex to acting on files with the specified metadata.


code:
More Windows porting.. Seem to be getting near an end of the easy stuff,
and also the webapp is getting pretty usable on Windows now, the only
really important thing lacking is XMPP support.

Made git-annex on Windows set HOME when it's not already set. Several of
the bundled cygwin tools only look at HOME. This was made a lot harder and
uglier due to there not being any way to modify the environment of the
running process.. git-annex has to re-run itself with the fixed
environment.

Got rsync.net working in the webapp. Although with an extra rsync.net
password prompt on Windows, which I cannot find a way to avoid.

While testing that, I discovered that openssh 6.5p1 has broken support for
~/.ssh/config Host lines that contain upper case letters! I have filed a
bug about this and put a quick fix in git-annex, which sometimes generated
such lines.


code:
Working on building [[design/metadata]] filtered branches.

Spent most of the day on types and pure code. Finally at the end
I wrote down two actions that I still need to implement to make
it all work:

[[!format haskell """
applyView' :: MkFileView -> View -> Annex Git.Branch
updateView :: View -> Git.Ref -> Git.Ref -> Annex Git.Branch
"""]]

I know how to implement these, more or less. And in most cases
they will be pretty fast.

The more interesting part is already done. That was the issue of how to
generate filenames in the filter branches. That depends on the `View` being
used to filter and organize the branch, but also on the original filename used
in the reference branch. Each filter branch has a reference branch (such as
"master"), and displays a filtered and metadata-driven reorganized tree
of files from its reference branch.

[[!format haskell """
fileViews :: View -> (FilePath -> FileView) -> FilePath -> MetaData -> Maybe [FileView]
"""]]

So, a view that matches files tagged "haskell" or "git-annex"
and with an author of "J\*" will generate filenames like
"haskell/Joachim/interesting_theoretical_talk.ogg" and
"git-annex/Joey/mytalk.ogg".

It can also work backwards from these
filenames to derive the MetaData that is encoded in them.

[[!format haskell """
fromView :: View -> FileView -> MetaData
"""]]

So, copying a file to "haskell/Joey/mytalk.ogg" lets it know that
it's gained a "haskell" tag. I knew I was on the right track when
`fromView` turned out to be only 6 lines of code!

The trickiest part of all this, which I spent most of yesterday thinking
about, is what to do if the master branch has files in subdirectories. It
probably does not makes sense to retain that hierarchical directory
structure in the filtered branch, because we instead have a
non-hierarchical metadata structure to express. (And there would probably
be a lot of deep directory structures containing only one file.) But
throwing away the subdirectory information entirely means that two files
with the same basename and same metadata would have colliding names.

I eventually decided to embed the subdirectory information into the filenames
used on the filter branch. Currently that is done by converting
`dir/subdir/file.foo` to `file(dir)(subdir).foo`. We'll see how this works
out in practice..


code:
Today I built `git annex view`, and `git annex vadd` and a few related
commands. A quick demo:

<pre>
joey@darkstar:~/lib/talks>ls
Chaos_Communication_Congress/  FOSDEM/       Linux_Conference_Australia/
Debian/                        LibrePlanet/  README.md
joey@darkstar:~/lib/talks>git annex view tag=*
view  (searching...)
Switched to branch 'views/_'
ok
joey@darkstar:~/lib/talks#_>tree -d
.
|-- Debian
|-- android
|-- bigpicture
|-- debhelper
|-- git
|-- git-annex
`-- seen

7 directories
joey@darkstar:~/lib/talks#_>git annex vadd author=*
vadd
Switched to branch 'views/author=_;_'
ok
joey@darkstar:~/lib/talks#author=_;_>tree -d
.
|-- Benjamin Mako Hill
|   `-- bigpicture
|-- Denis Carikli
|   `-- android
|-- Joey Hess
|   |-- Debian
|   |-- bigpicture
|   |-- debhelper
|   |-- git
|   `-- git-annex
|-- Richard Hartmann
|   |-- git
|   `-- git-annex
`-- Stefano Zacchiroli
    `-- Debian

15 directories
joey@darkstar:~/lib/talks#author=_;_>git annex vpop
vpop 1
Switched to branch 'views/_'
ok
joey@darkstar:~/lib/talks#_>git annex vadd tag=git-annex
vadd
Switched to branch 'views/(git-annex)'
ok
joey@darkstar:~/lib/talks#(git-annex)>ls
1025_gitify_your_life_{Debian;2013;DebConf13;high}.ogv@
git_annex___manage_files_with_git__without_checking_their_contents_into_git_{FOSDEM;2012;lightningtalks}.webm@
mirror.linux.org.au_linux.conf.au_2013_mp4_gitannex_{Linux_Conference_Australia;2013}.mp4@
joey@darkstar:~/lib/talks#_>git annex vpop 2
vpop 2
Switched to branch 'master'
ok
</pre>

Not 100% happy with the speed -- the generation of the view branch is close
to optimal, and fast enough (unless the branch has very many matching
files). And `vadd` can be quite fast if the view has already limited the
total number of files to a smallish amount. But `view` has to look at every
file's metadata, and this can take a while in a large repository. Needs indexes.

It also needs integration with `git annex sync`, so the view branches
update when files are added to the master branch, and moving files around
inside a view and committing them does not yet update their metadata.

---

Today's work was sponsored by Daniel Atlas.


code:
Still working on views. The most important addition today is that
`git annex precommit` notices when files have been moved/copied/deleted
in a view, and updates the metadata to reflect the changes.

Also wrote some walkthrough documentation: [[tips/metadata_driven_views]].
And, recorded a screencast demoing views, which I will upload next time
I have bandwidth.


code:
Spent the day catching up on the last week or so's traffic. Ended up
making numerous small big fixes and improvements. Message backlog stands at
44.

Here's the [[screencast demoing views|videos/git-annex_views_demo]]!

Added to the design today the idea of
automatically deriving metadata from the location of files in the master
branch's directory tree. Eg, `git annex view tag=* podcasts/=*` in a
repository that has a `podcasts/` directory would make a tree like
"$tag/$podcast". Seems promising.

So much still to do with views.. I have belatedly added them to
the roadmap for this month; doing Windows and Android in the same month was
too much to expect.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlJEI45rGczFAnuM7gRSj4C6s9AS9yPZDc"
 nickname="Kevin"
 subject="Neat!"
 date="2014-02-20T21:59:05Z"
 content="""
When the [[metadata design|day_112__metadata_design]] stuff appeared on the blog I didn't understand what you meant by automatically creating new tree layouts.  I'm really liking these views and can already imagine how useful it would be to tag my photos by person/place/time.  This is awesome!  Keep up the good work.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawneiQ3iR9VXOPEP34u7m_L3Qr28H1nEfE0"
 nickname="Ethan"
 subject="LFS"
 date="2014-02-21T00:03:59Z"
 content="""
You might be interested in the Logic File System at http://www.padator.org/wiki/wiki-LFS/doku.php or http://www.padator.org/papers/ which has a similar idea with views and metadata.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmZgZuUhZlHpd_AbbcixY0QQiutb2I7GWY"
 nickname="Jimmy"
 subject="comment 3"
 date="2014-02-21T07:05:34Z"
 content="""
I agree with Kevin as to the potential usefulness for photos.  Particularly if there's some way of automatically extracting and using tags or other EXIF metadata.
"""]]