code:
Had to toss out my XMPP presence hack. Turns out that, at least in Google
Talk, presence info is not sent to clients that have marked themselves
unavailable, and that means the assistant would not see notifications, as it
was nearly always marked unavailable as part of the hack.

I tried writing a test program that uses XMPP personal eventing, only
to find that Google Talk rejected my messages. I'm not 100% sure my
messages were right, but I was directly copying the example in the RFC,
and prosody accepted them. I could not seem to get a list of extensions out
of Google Talk either, so I don't know if it doesn't support personal
eventing, or perhaps only supports certian specific types of events.

So, plan C... using XMPP [presence extended content](http://xmpp.org/rfcs/rfc6121.html#presence-extended).
The assistant generates a presence message tagged "xa" (Extended Away),
which hopefully will make it not seem present to clients.
And to that presence message, I add my own XML element:

	<git-annex xmlns='git-annex' push="uuid,uuid" />

This is all entirely legal, and not at all a hack.
(Aside from this not really being presence info.) Isn't XML fun?

And plan C works, with Google Talk, and prosody. I've successfully gotten
push notifications flowing over XMPP!

----

Spent some hours dealing with an unusual probolem: git-annex started
segfaulting intermittently on startup with the new XMPP code.

Haskell code is not supposed to segfault..

I think this was probably due to not using a bound thread for XMPP,
so if haskell's runtime system recheduled its green thread onto a different
OS thread during startup, when it's setting up TLS, it'd make gnuTLS very
unhappy.

So, fixed it to use a bound thread. Will wait and see if the crash is gone.

----

Re-enabled DBUS support, using a new version of the library that avoids the
memory leak. Will need further changes to the library to support
reconnecting to dbus.

----

Next will be a webapp configuration UI for XMPP. Various parts of the
webapp will direct the user to set up XMPP, when appropriate, especially
when the user sets up a cloud remote.

To make XMPP sufficiently easy to configure, I need to check SRV records to
find the XMPP server, which is an unexpected PITA because `getaddrinfo`
can't do that. There are several haskell DNS libraries that I could use for
SRV, or I could use the `host` command:
`host -t SRV _xmpp-client._tcp.gmail.com`


code:
[[!comment format=mdwn
 username="http://phil.0x539.de/"
 nickname="Philipp Kern"
 subject="How to deal with offline messages being eaten?"
 date="2012-11-09T14:05:30Z"
 content="""
How do you avoid eating up normal text messages when git-annex is the only client left? Granted, Google Talk does some additional magic so that it shows you the history when connecting with the web page client. But with other servers it would be inconvenient if new messages weren't saved as offline messages but sent to the dæmon instead.
"""]]

repo_name: kakafarm/git

license: unknown


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.27"
 subject="comment 2"
 date="2012-11-09T21:55:42Z"
 content="""
It does try to mark itself as extended away, but yes, I think this is a potential concern. If you find the assistant interacts badly with your other clients, you can certainly give it its own XMPP account.
"""]]


code:
[[!comment format=mdwn
 username="http://phil.0x539.de/"
 nickname="Philipp Kern"
 subject="comment 3"
 date="2012-11-10T18:50:46Z"
 content="""
Sure, I could, as I'm operating my own server anyway. Others might not be willing to go to some random server and create another account, though.

Reviewing RFC 6121: Did you try negative priorities for your resources already? It's possible that Gtalk does something weird but in theory they should be ignored for messages directed to non-qualified JIDs (i.e. without an explicit resource setting). Setting \"xa\" alone won't help you anything, it will only cause the others to see you as \"xa\" but that's a perfectly valid common chat status where you expect messages to be queued in the client until the user returns.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.27"
 subject="comment 4"
 date="2012-11-10T19:48:01Z"
 content="""
I hadn't tried it, but it looks like a very good idea. I will have to change my protocol a little bit as I currently sometimes send chat messages not directed to a specific client, in order to reach any clients using an account. To reach clients with a negative priority, chat messages have to be directed at the client.

Ok, done; priority is set to -1 and all the XMPP messaging in git-annex still seems to work!

I don't know though, that this entirely solves the problem. In particular, a regular xmpp client may decide to direct a chat message at a git-annex client, due to that being the only client connected. If this does become a problem, I suppose git-annex could buffer and re-send such messages when it sees a regular client connect. Or it could show them in the webapp, but that seems feature creep.
"""]]


code:
Built a SRV lookup library that can use either `host` or ADNS.

Worked on DBUS reconnection some more; found a FD leak in the dbus library,
and wrote its long-suffering author, John Millikin (also the XMPP library
author, so I've been bothering him a lot lately), who once again came
through with a quick fix.

Built a XMPP configuration form, that tests the connection to the server.
Getting the wording right on this was hard, and it's probably still not
100% right.

[[!img /assistant/xmpp.png]]

Pairing over XMPP is something I'm still thinking about. It's
contingent on tunneling git over XMPP (actually not too hard),
and getting a really secure XMPP connection (needs library improvements,
as the library currently accepts any SSL certificate).


code:
Continuing to flail away at this XMPP segfault, which turned out not to be
fixed by bound threads. I managed to make a fairly self-contained and small
reproducible test case for it that does not depend on the network.
Seems the bug is gonna be either in the Haskell binding for GNUTLS,
or possibly in GNUTLS itself.

Update: John was able to fix it using my testcase! It was a GNUTLS
credentials object that went out of scope and got garbage collected.
I think I was seeing the crash only with the threaded runtime because
it has a separate garbage collection thread.

----

Arranged for the XMPP thread to restart when network connections
change, as well as when the webapp configures it.

Added an alert to nudge users to enable XMPP. It's displayed after adding a
remote in the cloud.

[[!img /assistant/xmppnudge.png]]

----

So, the first stage of XMPP is done. But so far all it does is push
notification. Much more work to do here.


code:
Back in [[day_85__more_foundation_work]], I wrote:

> I suspect, but have not proven,
> that the assistant is able to keep repos arranged in any shape of graph in
> sync, as long as it's connected (of course) and each connection is
> bi-directional. [And each node is running the assistant.]

After today's work, many more graph topologies can be kept in sync -- the
assistant now can keep repos in sync that are not directly connected, but
must go through a central transfer point, which does not run the assistant
at all. Major milestone!

To get that working, as well as using XMPP push notifications, it turned
out to need to be more agressive about pushing out changed location log
information. And, it seems, that was the last piece that was missing.
Although I narrowly avoided going down a blind alley involving sending
transfer notifications over XMPP. Luckily, I came to my senses.

----

This month's focus was the cloud, and the month is almost done. And now
the assistant can, indeed be used to sync over the cloud! I would have
liked to have gotten on to implementing Amazon Glacier or Google Drive
support, but at least the cloud fundamentals are there.

Now that I have XMPP support, I'm tending toward going ahead and adding
XMPP pairing, and git push over XMPP. This will open up lots of excellent
use cases.

So, how to tunnel git pushes over XMPP? Well, `GIT_SHELL` can be set to
something that intercepts the output of `git-send-pack` and
`git-receive-pack`, and that data can be tunneled through XMPP to connect
them. Probably using XMPP ping.
(XEP-0047: In-Band Bytestreams would be the right way ...
but of course Google Talk doesn't support that extension.)

XMPP requires ugly encoding that will bloat the data, but the data
quantities are fairly small to sync up a few added or moved files
(of course, we'll not be sending file contents over XMPP). Pairing with
an large git repository over XMPP will need rather more bandwidth,
of course.


code:
Spent most of the past day moving the assistant into a monad of its own
that encapsulates all the communications channels for its threads. This
involved modifiying nearly every line of code in the whole assistant.

Typical change:

[[!format haskell """
handleConnection threadname st dstatus scanremotes pushnotifier = do
        reconnectRemotes threadname st dstatus scanremotes (Just pushnotifier)
                =<< networkRemotes st

handleConnection = reconnectRemotes True =<< networkRemotes
"""]]

So, it's getting more readable..


code:
Finished working the new assistant monad into all the assistant's code.
I've changed 1870 lines of code in the past two days. It feels like more.
While the total number of lines of code has gone up by around 100, the
actual code size has gone *down*; the monad allowed dropping 3.4 kilobytes
of manual variable threading complications. Or around 1% of a novel edited
away, in other words.

I don't seem to have broken anything, but I'm started an extensive test
of all the assistant and webapp. So far, the bugs I've found were not
introduced by my monadic changes. Fixed several bugs around adding
removable drives, and a few other minor bugs. Plan to continue testing
tomorrow.


code:
I've been investigating how to make `git annex watch` work on
FreeBSD, and by extension, OSX.

One option is kqueue, which works on both operating systems, and allows
very basic monitoring of file changes. There's also an OSX specific
hfsevents interface.

Kqueue is far from optimal for `git annex watch`, because it provides even
less information than inotify (which didn't really provide everything I
needed, thus the lsof hack). Kqueue doesn't have events for files being
closed, only an event when a file is created. So it will be difficult for
`git annex watch` to know when a file is done being written to and can be
annexed. git annex will probably need to run lsof periodically to check when
recently added files are complete. (hsevents shares this limitation)

Kqueue also doesn't provide specific events when a file or directory is
moved. Indeed, it doesn't provide specific events about what changed at
all. All you get with kqueue is a generic "oh hey, the directory you're
watching changed in some way", and it's up to you to scan it to work out
how. So git annex will probably need to run `git ls-tree --others`
to find changes in the directory tree. This could be expensive with large
trees. (hsevents has per-file events on current versions of OSX)

Despite these warts, I want to try kqueue first, since it's more portable
than hfsevents, and will surely be easier for me to develop support for,
since I don't have direct access to OSX.

So I went to a handy Debian kFreeBSD porter box, and tried some kqueue
stuff to get a feel for it. I got a python program that does basic
directory monitoring with kqueue to work, so I know it's usable there.

Next step was getting kqueue working from Haskell. Should be easy, there's
a Haskell library already. I spent a while trying to get it to work on
Debian kFreeBSD, but ran into a
[problem](https://github.com/hesselink/kqueue/issues/1) that could be
caused by the Debian kFreeBSD being different, or just a bug in the Haskell
library. I didn't want to spend too long shaving this yak; I might install
"real" FreeBSD on a spare laptop and try to get it working there instead.

But for now, I've dropped down to C instead, and have a simple C program
that can monitor a directory with kqueue. Next I'll turn it into a simple
library, which can easily be linked into my Haskell code. The Haskell code
will pass it a set of open directory descriptors, and it'll return the
one that it gets an event on. This is necessary because kqueue doesn't
recurse into subdirectories on its own.

I've generally had good luck with this approach to adding stuff in Haskell;
rather than writing a bit-banging and structure packing low level interface
in Haskell, write it in C, with a simpler interface between C and
Haskell.


code:
Did a lot of testing, found and fixed 4 bugs with repository setup
configurators. None of them were caused by the recent code reworking.


code:
Got the XMPP client maintaining a list of buddies, including tracking which
clients are present and away, and which clients are recognised as other
git-annex assistant clients. This was fun, it is almost all pure
functional code, which always makes me happy.

Started building UI for XMPP pairing. So far, I have it showing a list of
buddies who are also running git-annex (or not). The list even refreshes
in real time as new buddies come online.

[[!img /assistant/buddylist.png]]


code:
Reworked my XMPP code, which was still specific to push notification, into
a more generic XMPP client, that's based on a very generic NetMessager
class, that the rest of the assistant can access without knowing anything
about XMPP.

Got pair requests flowing via XMPP ping, over Google Talk! And when the
webapp receives a pair request, it'll pop up an alert to respond. The rest
of XMPP pairing should be easy to fill in from here.

To finish XMPP pairing, I'll need git pull over XMPP, which is nontrivial,
but I think I know basically how to do. And I'll need some way to represent
an XMPP buddy as a git remote, which is all that XMPP pairing will really
set up.

It could be a git remote using an `xmpp:user@host` URI for the git url, but
that would confuse regular git to no end (it'd think it was a ssh host),
and probably need lots of special casing in the parts of git-annex that
handle git urls too. Or it could be a git remote without an url set, and
use another config field to represent the XMPP data. But then git wouldn't
think it was a remote at all, which would prevent using "git pull
xmppremote" at all, which I'd like to be able to use when implementing git
pull over XMPP.

Aha! The trick seems to be to leave the url unset in git config,
but temporarily set it when pulling:

	GIT_SSH=git-annex git git -c remote.xmppremote.url=xmpp:client pull xmppremote

Runs git-annex with "xmpp git-upload-pack 'client'".. Just what I need.


code:
[[!comment format=mdwn
 username="http://meep.pl/"
 ip="193.219.28.34"
 subject="git-remote-helpers"
 date="2012-11-09T08:54:11Z"
 content="""
It seems that xmpp::user@host (with double colon) would cause git to try to use a remote-xmpp helper.
Wouldn't it be a way to have the remote address (if not really URL) set and not be treated as ssh?

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.27"
 subject="comment 2"
 date="2012-11-09T17:41:08Z"
 content="""
Thanks, I think that's exactly what I should do.
"""]]


code:
Spent about 5 hours the other night in XMPP hell. At every turn Google Talk
exhibited behavior that may meet the letter of the XMPP spec (or not), but
varies between highly annoying and insane.

By "insane", I mean this: If a presence message is directed from one client
to another client belonging to that same user, randomly leaking that message
out to other users who are subscribed is just a security hole waiting to
happen.

Anyway, I came out of that with a collection of hacks that worked, but I
didn't like. I was using directed presence for buddy-to-buddy pairing, and
an IQ message hack for client-to-client pairing.

Today I got chat messages working instead, for both sorts of pairing. These
chat messages have an empty body, which *should* prevent clients from
displaying them, but they're sent directed to only git-annex clients
anyway.

----

And XMPP pairing 100% works now! Of course, it doesn't know how to git pull
over XMPP yet, but everything else works.

Here's a real `.git/config` generated by the assistant after XMPP pairing.

	[remote "joey"]
	        url =
	        fetch = +refs/heads/*:refs/remotes/joey/*
	        annex-uuid = 14f5e93e-1ed0-11e2-aa1c-f7a45e662d39
	        annex-xmppaddress = joey@kitenet.net

----

Fixed a typo that led to an infinite loop when adding a ssh git repo with the
assistant. Only occurred when an absolute directory was specified, which
is why I didn't notice it before.

----

Security fix: Added a `GIT_ANNEX_SHELL_DIRECTORY` environment variable that
locks down git-annex-shell to operating in only a single directory. The
assistant sets that in ssh `authorized_keys` lines it creates. This
prevents someone you pair with from being able to access any other git or
git-annex repositories you may have.

----

Next up, more craziness. But tomorrow is Nov 6th, so you in the US already
knew that..


code:
Laying the groundwork for git push over XMPP. BTW, the motivation for doing
this now is that if the assistant can push git data peer-to-peer, users
who are entirely using the cloud don't need to set up a git repo in the
cloud somewhere. Instead, a single special remote in the cloud will be all
they need. So this is a keystone in the assistant's cloud support.

I'm building the following pipeline:

	git push <--> git-annex xmppgit <--> xmppPush <-------> xmpp
	                                                         |
	git receive-pack <--> xmppReceivePack <---------------> xmpp

A tricky part of this is `git-annex xmppgit`, which is run by `git push`
rather than the usual `ssh`. Rather than speak XMPP itself, that feeds the
data through the assistant daemon, using some special FDs that are set
up by the assistant when it runs `git push`, and communicated via
environment variables. I hoped to set up a pipe and not need it to do any
work on its own, but short of using the linux-specific `splice(2)`, that
doesn't seem possible. It also will receive the exit status of
`git receive-pack` and propigate it to `git push`.

Also built the IO sides of `xmppPush` and `xmppReceivePack` although these
are not tested. The XMPP sides of them come next.

----

Stuffing lots of git-annex branded USB keys into envelopes tonight, while
watching the election coverage.


code:
I've finished building the XMMP side of git push over XMPP. Now I only
have to add code to trigger these pushes. And of course, fix all the bugs,
since none of this has been tested at all.

Had to deal with some complications, like handling multiple clients that
all want to push at the same time. Only one push is handled at a time;
messages for others are queued. Another complication I don't deal with yet
is what to do if a client stops responding in the middle of a push. It
currently will wait forever for a message from the client; instead it
should time out.

----

Jimmy got the OSX builder working again, despite my best attempts to add
dependencies and break it.


code:
I'm stunned and stoked to have gotten git push over XMPP working today.
And am nearly out of steam, it was a wild ride..

	To xmpp::joey@kitenet.net
	  * [new branch]      master -> refs/xmpp/newmaster

The surprising part is how close my initial implementation came to just
working on the first try. It had around 3 bugs, which took hours of staring
at debugging output to find:

1. The git push action was run in the same thread as the XMPP
   client, which prevented the client from continuing to run and relaying
   messages.
2. The git-receive-pack side waited on the wrong thread, so didn't
   notice when the program was done.
3. I accidentially used the wrong attribute name when sending a ReceivePackDone
   message.

But all in all, it just worked.

Here's a sample of the actual data sent when one file is added to the
repository (also includes the corresponding update to the git-annex branch):

	MDA4NjhhMmNmOGZjMWE3MTlkOGVjOWVmOWZiMGZiNjVlODc2NjQ1NDAyMTAgODIwNTZjMDM4
	ZjU2YzE1ODdjYzllOWRhNzQzMzU0YjE4NzNjZWJlOSByZWZzL3htcHAvbmV3bWFzdGVyACBy
	ZXBvcnQtc3RhdHVzIHNpZGUtYmFuZC02NGswMDAw

	UEFDSwAAAAIAAAADnAx4nJXLTQ4CIQxA4T2n4AKaAqVAYoxL4y2gU+Jo5iczdeHtnSu4eMm3
	ebqJ2NwgSCLmNkTBlKFCYwwhoHOtQ+scqZCwWesms9pcPffc2dXkypCFi/TSG/RGUXIiwojg
	HZj60eey2cciX3uXfbeX18Hbe1SZRc9HV+tC9FgyJW9PgACGl2kaVeXfz/wArHQ81qMGeJwz
	NDIAAoVUI4ZZB9RW1E8NtXp5t77/fn3hw41cl2MNIbIZqTk5+Qwerw+aJX2INjsffYndtdCz
	5mZWLDdUQV5qeVpmDtCQnx/3/6s40+Q4P/7O+Y4ShS+1Ad83AwC6CirftAt4nK3MsRGDMAwF
	0IkcSVgSdpkidzRUmcDWBy4pSAEFl+mzRN4A77a9Tmr7vlz06e8lzoPmmb5Mz+k+mD/SkTkl
	eFHPq9eqQ+nSzFsWaDFnFmCMCEOvHgLrCrQxS7AWdvUVhv9uPwHxMbfumlvWdco1RLL4wSQF
	g0uFFOKu3Q==

Git said this push took 385 bytes; after base64 encoding to transport it over
XMPP as shown above, it needs 701 bytes, and the XMPP envelope and encryption
adds more overhead (although the XMPP connection may also be compressed?)

Not the most efficient git transport, but still a practical one!

----

Big thanks by the way to meep, who posted a comment reminding me about
`git-remote-helpers`. This was the right thing to use for XMPP over git,
it lets the git remote be configured with `url = xmpp::user@host`.

----

Next, I need to get the assistant to use this for syncing. Currently, it only
pushes a test branch.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm5vDem1yeIu6uith5pxfb4mdKdIWVJpCs"
 nickname="Louis"
 subject="Base64 vs. yEnc"
 date="2012-11-12T02:02:17Z"
 content="""
Would yEnc be a suitable alternative to base64 for encoding the binary transfers over XMPP?  Or why not?

Thanks!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.27"
 subject="comment 2"
 date="2012-11-12T02:39:19Z"
 content="""
AFAIK, yenc can use xml characters like < and > in its encoding.
"""]]


code:
I got full-on git-annex assistant syncing going over XMPP today!

How well does it work? Well, I'm at the cabin behind a dialup modem. I have
two repos that can only communicate over XMPP. One uses my own XMPP server,
and the other uses a Google Talk account. I make a file in one repo, and
switch windows to the other, and type `ls`, and the file (not its content
tho..) has often already shown up. So, it's about as fast as syncing over
ssh, although YMMV.

----

Refactored the git push over XMPP code rather severely. It's quite a
lot cleaner now.

----

Set XMPP presence priority to a negative value, which will hopefully
prevent git-annex clients that share a XMPP account with other clients from
intercepting chat messages. Had to change my XMPP protocol some to deal
with this.

----

Some webapp UI work. When showing the buddy list, indicate which buddies
are already paired with.

After XMPP pairing, it now encourages setting up a shared cloud repository.

[[!img /assistant/xmpppairingend.png]]

I still need to do more with the UI after XMPP pairing, to help the paired
users configure a shared cloud transfer remote. Perhaps the thing to do is
for the ConfigMonitor to notice when a git push adds a new remote,
and pop up an alert suggesting the user enable it. Then one user
can create the repository, and the other one enable it.


code:
I hope I'm nearly at the end of this XMPP stuff after today. Planning a new
release tomorrow.

----

Split up the local pairing and XMPP pairing UIs, and wrote a
[[/assistant/share_with_a_friend_walkthrough]].

----

Got the XMPP push code to time out if expected data doesn't arrive within
2 minutes, rather than potentially blocking other XMPP push forever if
the other end went away.

I pulled in the Haskell
[async](http://hackage.haskell.org/package/async) library for this,
which is yes, yet another library, but one that's now in the haskell platform.
It's worth it, because of how nicely it let me implement IO actions that
time out.

[[!format haskell """
runTimeout :: Seconds -> IO a -> IO (Either SomeException a)
runTimeout secs a = do
        runner <- async a
        controller <- async $ do
                threadDelaySeconds secs
                cancel runner
        cancel controller `after` waitCatch runner
"""]]

This would have been 20-50 lines of gnarly code without async, and I'm sure
I'll find more uses for async in the future.

----

Discovered that the XMPP push code could deadlock, if both clients started
a push to the other at the same time. I decided to fix this by allowing
each client to run both one push and one receive-pack over XMPP at the same
time.

----

Prevented the transfer scanner from trying to queue transfers to XMPP remotes.

----

Made XMPP pair requests that come from the same account we've already
paired with be automatically accepted. So once you pair with one device,
you can easily add more.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmqVi9eQjkZt8EC-byTXJ8TnY7VyOHzW2s"
 nickname="Zahary"
 subject="Pairing on the local network"
 date="2012-11-12T11:39:17Z"
 content="""
Joey, How does pairing on the local network work btw?
Now that you have experience intercepting and relaying the git pack output maybe you can use UDP broadcast on the local network for a truly fast sync.
With dropbox, I'm often syncing working copies of software projects between several VMs so I can test the code on all platforms before committing. I've found that to be faster (for builds) then using a VM shared folders, network shares, etc.
"""]]


code:
[[!comment format=mdwn
 username="http://phil.0x539.de/"
 nickname="Philipp Kern"
 subject="comment 2"
 date="2012-11-12T11:50:42Z"
 content="""
Developing something new based on IPv4 UDP broadcast seems to be insane. IPv6 link-local multicast should be available virtually anywhere. XMPP relies on TCP and a central server to guarantee that a single packet is not split up, which would need quite some protocol engineering to get right over lossy UDP (e.g. segmentation, flow control, congestion avoidance). But TCP as discovered using mDNS might work… Still needs some kind of authentication / encryption, though.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmqVi9eQjkZt8EC-byTXJ8TnY7VyOHzW2s"
 nickname="Zahary"
 subject="Jingle"
 date="2012-11-12T12:00:50Z"
 content="""
Also, you probably know about Jingle:
http://en.wikipedia.org/wiki/Jingle_(protocol)

What are you thoughts about it? I guess it would have been harder to use the Google C++ library from Haskell and full mesh peer-to-peer is certainly more complicated than using the XMPP server to effectively multicast to all paired clients.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmqVi9eQjkZt8EC-byTXJ8TnY7VyOHzW2s"
 nickname="Zahary"
 subject="@Philipp Kern"
 date="2012-11-12T12:13:33Z"
 content="""
Alright, I stand corrected, although I'm aware of some libraries that have already solved the problem so you don't really have to implement reliable UDP from scratch: http://code.google.com/p/openpgm/
"""]]


code:
[[!comment format=mdwn
 username="http://phil.0x539.de/"
 nickname="Philipp Kern"
 subject="comment 5"
 date="2012-11-12T12:50:26Z"
 content="""
To quote the website (emphasis mine):
> PGM is appropriate for applications that require duplicate-free multicast data delivery from multiple sources to multiple receivers. *PGM does not support acknowledged delivery, nor does it guarantee ordering of packets from multiple senders.*
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.27"
 subject="comment 6"
 date="2012-11-12T14:43:32Z"
 content="""
For local pairing, multicast UDP is used for discovery and authentication, and then all data transfers take place over ssh.
"""]]


code:
Cut a new release today. It's been nearly a month since the last one, and a
large number of improvements.. Be sure to read the
[[/assistant/release_notes]] if upgrading. All the standalone builds are
updated already.


code:
Followed my plan from yesterday, and wrote a simple C library to interface
to `kqueue`, and Haskell code to use that library. By now I think I
understand kqueue fairly well -- there are some very tricky parts to the
interface.

But... it still didn't work. After building all this, my code was
failing the same way that the
[haskell kqueue library failed](https://github.com/hesselink/kqueue/issues/1)
yesterday. I filed a [bug report with a testcase]().

Then I thought to ask on #haskell. Got sorted out in quick order! The
problem turns out to be that haskell's runtime has a periodic SIGALARM,
that is interrupting my kevent call. It can be worked around with `+RTS -V0`,
but I put in a fix to retry to kevent when it's interrupted.

And now `git-annex watch` can detect changes to directories on BSD and OSX!

Note: I said "detect", not "do something useful in response to". Getting
from the limited kqueue events to actually staging changes in the git repo
is going to be another day's work. Still, brave FreeBSD or OSX users
might want to check out the `watch` branch from git and see if
`git annex watch` will at least *say* it sees changes you make to your
repository.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 1"
 date="2012-06-19T06:53:26Z"
 content="""
heh, yea, it's detecting changes on OSX ;)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 1"
 date="2012-06-19T07:01:26Z"
 content="""
issues with the watch command on OSX, it seems that there is a race condition somewhere. I dumped a few iso's into an annex and it only annexed the smaller files (checksums) and the bigger ones (the iso's) just got made read only. also do you want these bugs to be logged here or in the bugs section?
"""]]


code:
Dealt with post-release feedback deluge. There are a couple weird bugs that
I don't understand yet. OSX app is still not working everywhere.

----

Got the list of repositories in the webapp to update automatically when
repositories are added, including when syncing with a remote causes
repositories to be discovered.

----

I need a plan for the rest of the month. It feels right to focus on more
cloud storage support. Particularly because all the cloud providers
supported so far are ones that, while often being best of breed, also cost
money. To finish up the cloud story, need support for some free ones.

Looking at the results of the [[polls/prioritizing_special_remotes]]
poll, I suspect that free storage is a large part of why Google Drive got
so many votes. Soo, since there is not yet a Haskell library for Google
Drive, rather than spending a large chunk of time writing one, I hope to
use a [Haskell WebDAV library](http://hackage.haskell.org/package/DAV)
that my friend Clint recently wrote. A generic
WebDAV special remote in git-annex will provide much better support for
box.com (which has 5 to 50 gb free storage), as well as all the
[OwnCloud providers](http://owncloud.org/providers/), at least one of which
provides 5 gb free storage.

If I have time left this month after doing that, I'd be inclined to do
Amazon Glacier. People have already gotten that working with git-annex, but
a proper special remote should be better/easier, and will allow integrating
support for it into the assistant, which should help deal with its long
retrieval delays. And since, if you have a lot of data
archived in Glacier, you will only want to pull out a few files at a time,
this is another place besides mobile phones where a [[partial_content]]
retrieval UI is needed. Which is on the roadmap to be worked on next
month-ish. Synergy, here I come. I hope.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm7AuSfii_tCkLyspL6Mr0ATlO6OxLNYOo"
 nickname="Georg"
 subject="definitely glacier would be cool"
 date="2012-11-14T08:51:40Z"
 content="""
Because having it available as an archival system to move stuff over that you don't expect to constantly need available. But what would be needed is a way to prevent the assistant to pull all the data from there back on your clients. This probably allready is doable with preferred content, but probably needs an easier way to set up - when I tried the new version it immediately started to pull all the stuff from my server repository that I moved there to get my disk space free. So my scenario would be \"move stuff to archives and when I drop it afterwards, keep it dropped locally if at least one archival server has it\".
"""]]


code:
Read up on WebDAV, and got the haskell library working. Several hours
were wasted by stumbling over a bug in the library, that requires a
carefully crafted XML document to prevent. Such a pity about things
like DAV (and XMPP) being designed back when people were gung-ho about
XML.. but we're stuck with them now.

Now I'm able to send and receive files to box.com using the library. Trying to
use an OwnCloud server, though, I get a most strange error message, which
looks to be coming from deep in the HTTPS library stack: "invalid IV length"

The haskell DAV library didn't have a way to delete files. I've added one
and sent off a patch.

Roughed in a skeleton of a webdav special remote. Doesn't do anything yet.
Will soon.

Factored out a Creds module from parts of the S3 special remote and XMPP
support, that all has to do with credentials storage. Using this for webdav
creds storage too.

Will also need to factor out the code that's currently in the directory
special remote, for chunking of files.

----

PS: WebDAV, for all its monstrously complicated feature set, lacks one obvious
feature: The ability to check how much free space is available to store
files. Eyeroll.


code:
Two releases of the Haskell DAV library today. First release had my changes
from yesterday. Then I realized I'd also need support for making WebDAV
"collections" (subdirectories), and sent Clint a patch for that too, as
well as a patch for querying DAV properties, and that was 0.2.
Got it into Debian unstable as well. Should have everything I'll need now.

The webdav special remote is now working! Still todo:
Encryption support, progress bars, large file chunking, and webapp
configurators. But already, it's a lot nicer than the old approach of using
davfs2, which was really flakey and slow with large data volumes.

I did notice, though, that uploading a 100 mb file made the process use 100
mb of memory. This is a problem I've struggled with earlier with S3, the
Haskell http libraries are prevented from streaming data by several parts
of the protocol that cause the data to be accessed more than once. I guess
this won't be a problem for DAV though, since it'll probably be chunking
files anyway.

---

Mailed out all my Kickstarter USB key rewards, and ordered
the T-shirts too.


code:
Worked on webdav special remotes all day.

* Got encryption working,
  after fixing an amusing typo that made `initremote` for webdav throw away the
  encryption configuration and store files unencrypted.
* Factored out parts of the directory special remote that had to do with file
  chunking, and am using that for webdav. This refactoring was painful.

At this point, I feel the webdav special remote works better than the old
davfs2 + directory special remote hack. While webdav doesn't yet have
progress info for uploads, that info was pretty busted anyway with
davfs2 due to how it buffers files. So ... I've merged webdav into master!

-----

Tomorrow, webapp configurators for Box.com and any other webdav supporting
sites I can turn up and get to work..

-----

A while ago I made git-annex not store login credentials in git for special
remotes, when it's only encrypting them with a shared cipher. The
rationalle was that you don't want to give everyone who gets ahold of your
git repo (which includes the encryption key) access to your passwords,
Amazon S3 account, or whatever. I'm now considering adding a checkbox (or
command-line flag) that allows storing the login credentials in git,
if the user wants to. While using public key crypto is the real solution
(and is fully supported by git-annex (but not yet configurable in the
webapp)), this seems like a reasonable thing to do in some circumstances,
like when you have a Box.com account you really do want to share with
the people who use the git repo.


code:
I needed an easy day, and I got one. Configurator in the webapp for Box.com
came together really quickly and easily, and worked on the first try.

Also filed a [bug](https://github.com/vincenthz/hs-cryptocipher/issues/21)
on the Haskell library that is failing on portknox.com's SSL certificate.
That site is the only OwnCloud provider currently offering free WebDAV
storage. Will hold off on adding OwnCloud to the webapp's cloud provider lists
until that's fixed.


code:
Unexpectedly today, I got progress displays working for uploads via WebDAV.

The roadblock had been that the interface of for uploading to S3 and WebDAV
is something like `ByteString -> IO a`. Which doesn't provide any hook to
update a progress display as the ByteString is consumed.

My solution to this was to create a `hGetContentsObserved`, that's similar
to `hGetContents`, but after reading each 64kb chunk of data from the
Handle to populate the ByteString, it runs some observing action. So when
the ByteString is streamed, as each chunk is consumed, the observer
runs. I was able to get this to run in constant space, despite not having
access to some of the ByteString internals that `hGetContents` is built
with.

So, a little scary, but nice. I am curious if there's not a better way
to solve this problem hidden in a library somewhere. Perhaps it's another
thing that conduit solves somehow? Because if there's not, my solution
probably deserves to be put into a library. Any Haskell folk know?

----

Used above to do progress displays for uploads to S3. Also did progress
display to console on download from S3. Now very close to being done
with [[progressbars]]. Finally. Only bup and hook remotes need progress
work.

----

Reworked the core crypto interface, to better support streaming data through
gpg. This allowed fixing both the directory and webdav special remotes to
not buffer whole files in memory when retrieving them as chunks from the
remote.

-----

Spent some time dealing with API changes in Yesod and Conduit. Some of them
annoyingly gratuitous.


code:
Changed how the directory and webdav special remotes store content.
The new method uses less system calls, is more robust, and leaves any
partially transferred key content in a single tmp directory, which
will make it easier to clean that out later.

Also found & fixed a cute bug in the directory special remote when the
chunksize is set to a smaller value than the ByteString chunk size, that
caused it to loop forever creating empty files.

----

Added an embedcreds=yes option when setting up special remotes.
Will put UI for it into the webapp later, but rather than work on that
tomorrow, I plan to work on glacier.


code:
Got Amazon Glacier working as a full-fledged special remote.

(Well, I think it works... Since it takes 4 hours to get data out,
which is longer than the time it took me to sign up for Glacier and
write the special remote ... I've yet to fully test it!)

Thanks to Robie Basak for writing glacier-cli, and developing the intial
hook remote support. Also thanks to Peter Todd for pointing out that
Glacier cannot store empty files, which had to be worked around in the
special remote.

Of course the 4 hour delay on retreval makes Glacier interesting. For now,
you have to run "git annex get" twice, once to queue the retrieval, and a
second time in 4 hours to get the file(s). There is a helpful example in
[[tips/using_Amazon_Glacier]].

The real complication though, is that Glacier's inventories take a long
time to get, and can be out of date. So glacier-cli caches inventory info.
I didn't feel comfortable making git-annex trust that information,
so it'll refuse to trust that Glacier has a copy of a file when dropping
it. There's a `--trust-glacier` switch to override this default paranoid
behavior when dropping files.

----

Tomorrow ... er, tomorrow is Thanksgiving trip start.

Next weekend: Webapp configurator for glacier, and maybe something
to get the assistant to detect when jobs are complete and finish
retrievals from Glacier, automatically.


code:
Added a configurator for Glacier repositories to the webapp. That was the last
cloud repository configurator that was listed in the webapp and wasn't
done. Indeed, just two more repository configurators remain to be filled in:
phone and NAS.

By default, Glacier repositories are put in a new "small archive" group.
This makes only files placed in "archive" directories be sent to Glacier
(as well as being dropped from clients), unlike the full archive group
which archives all files. Of course you can change this setting, but
avoiding syncing all data to Glacier seemed like a good default, especially
since some are still worried about Glacier's pricing model.

Fixed several bugs in the handling of archive directories, and
the webapp makes a toplevel archive directory when an archive remote is
created, so the user can get on with using it.

Made the assistant able to drop local files immediately after transferring
them to glacier, despite not being able to trust glacier's inventory.
This was accomplished by making the transferrer, after a successful upload,
indicate that it trusts the remote it just uploaded to has the file,
when it checks if the file should be dropped.

Only thing left to do for glacier is to make the assistant retry failed
downloads from it after 4 hours, or better, as soon as they become
available.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmnG4EuvZWse5hvgrl0XAK-U61e-0iGaao"
 nickname="David"
 subject="NAS?"
 date="2012-11-25T12:27:50Z"
 content="""
What are you planning to support with the NAS repository type?  Will the assistant sync over NFS? I wanted to read more about this but I couldn't find anything on the design pages.
"""]]


code:
Got progress bars working for glacier. This needed some glacier-cli
changes, which Robie helpfully made earlier.

Spent some hours getting caught up and responding to bug reports, etc.

Spent a while trying to make git-annex commands that fail to find
any matching files to act on print a useful warning message,
rather than the current nothing. Concluded this will be surprisingly
hard to do, due to the multiple seek passes some commands perform. Update:
Thought of a non-invasive and not too ugly way to do this while on my
evening walk, and this wart is gone.


code:
Good news! My beta testers report that the new kqueue code works on OSX.
At least "works" as well as it does on Debian kFreeBSD. My crazy
development strategy of developing on Debian kFreeBSD while targeting Mac
OSX is vindicated. ;-)

So, I've been beating the kqueue code into shape for the last 12 hours,
minus a few hours sleep.

First, I noticed it was seeming to starve the other threads. I'm using
Haskell's non-threaded runtime, which does cooperative multitasking between
threads, and my C code was never returning to let the other threads run.
Changed that around, so the C code runs until SIGALARMed, and then that
thread calls `yield` before looping back into the C code. Wow, cooperative
multitasking.. I last dealt with that when programming for Windows 3.1!
(Should try to use Haskell's -threaded runtime sometime, but git-annex
doesn't work under it, and I have not tried to figure out why not.)

Then I made a [single commit](http://source.git-annex.branchable.com/?p=source.git;a=commitdiff;h=2bfcc0b09c5dd37c5e0ab65cb089232bfcc31934),
with no testing, in which I made the kqueue code maintain a cache of what
it expects in the directory tree, and use that to determine what files
changed how when a change is detected. Serious code. It worked on the
first go. If you were wondering why I'm writing in Haskell ... yeah,
that's why.

And I've continued to hammer on the kqueue code, making lots of little
fixes, and at this point it seems *almost* able to handle the changes I
throw at it. It does have one big remaining problem; kqueue doesn't tell me
when a writer closes a file, so it will sometimes miss adding files. To fix
this, I'm going to need to make it maintain a queue of new files, and
periodically check them, with `lsof`, to see when they're done being
written to, and add them to the annex. So while a file is being written
to, `git annex watch` will have to wake up every second or so, and run
`lsof` ... and it'll take it at least 1 second to notice a file's complete.
Not ideal, but the best that can be managed with kqueue.


code:
New release today, the main improvements in this one being WebDAV,
Box.com, and Amazon glacier support. [[/assistant/release_notes]]

Collected together all the OSX problem reports into one place
at [[/assistant/OSX]], to make it easier to get an overview of them.

Did some testing of the OSX standalone app and found that it was missing
some libraries. It seems some new libraries it's using themselves depend on
other libraries, and `otool -L` doesn't recursively resolve this.

So I converted the simplistic shell script it was using to install
libraries into a haskell progream that recursively adds libraries until
there are no more to add. It's pulling in quite a lot more libraries now.
This may fix some of the problems that have been reported with the
standalone app; I don't really know since I can only do very limited
testing on OSX.

Still working on getting the standalone builds for this release done,
should be done by the end of today.

Also found a real stinker of a bug in `dirContentsRecursive`, which was
just completely broken, apparently since day 1. Fixing that has certainly
fixed buggy behavior of `git annex import`. It seems that the other
user of it, the transfer log code, luckily avoided the deep directory
trees that triggered the bug.


code:
I had planned to do nothing today; I can't remember the last time I did
that. Twas not to be; instead I had to make a new release to fix
a utterly stupid typo in the rsync special remote. I'm also seeing
some slightly encouraging signs of the OSX app being closer to working
and this release has a further fix toward that end; unsetting all the
environment variables before running the system's web browser.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnbBRfl5F8gKRr1ko8Ai6FbEZStXXNF1S4"
 nickname="Áron"
 subject="deb package for amd64"
 date="2012-12-04T08:51:55Z"
 content="""
Hey, great work!
Will we also have 64bit binaries in debian unstable soon?

"""]]


code:
Just filling in a few remaining bits and pieces from this month's work.

* Made the assistant periodically check glacier-cli for archives that are
  ready, and queue downloads of them.
* The box.com configurator defaults to embedding the account info, allowing
  one-click enabling of the repository. There's a check box to control this.
* Fix some bugs with how the standalone images run git-annex.
* Included ssh in the standalone images.
* Various other bug fies.


code:
Yesterday, I woke up and realized I didn't know what I needed to work on in
git-annex. Added a poll, [[polls/Android]] to help me decide what major
thing to work on next.

----

More flailing at the OSX monster. (A species of gelatinous cube?)
Current fun seems to involve git processes spinning if git-annex
was started without a controlling TTY. I'm befuddled by this.

----

Made the S3 and Glacier configurators have a list of regions, rather than
requiring a the region's code be entered correctly. I could not find a list of
regions, or better, an API to get a list, so I'll have to keep updating as
Amazon adds services in new regions.

----

Spent some time trying to get WebDAV to work with livedrive.com.
It doesn't like empty PROPPATCH. I've developed a change to the haskell DAV
library that will let me avoid this problem.


code:
[[!comment format=mdwn
 username="http://1id.com/contact/=tp"
 nickname="Tom"
 subject="In case it might be helpful"
 date="2012-12-04T14:25:14Z"
 content="""
S3 and Glacier regions.
<http://docs.amazonwebservices.com/general/latest/gr/rande.html#s3_region>
<http://docs.amazonwebservices.com/general/latest/gr/rande.html#glacier_region>

(EC2) Region list API - likely a superset of the other service regions, could be used for sanity checks.
<http://docs.amazonwebservices.com/AWSEC2/latest/APIReference/ApiReference-query-DescribeRegions.html>
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm5vDem1yeIu6uith5pxfb4mdKdIWVJpCs"
 nickname="Louis"
 subject="AWS regions in JSON"
 date="2012-12-05T03:27:52Z"
 content="""
Mitch Garnaat maintains a json data file with AWS service regions & other useful data...

https://github.com/garnaat/missingcloud/blob/master/aws.json
"""]]


code:
Made the webapp show runtime errors on a prettified page that includes version
info, a bug reporting link, etc.

Dealt with a bad interaction between required fields and the bootstrap modals
displayed when submitting some configuration forms. This was long, complex,
and had lots of blind alleys. In the end, I had to derive new password and
text fields in yesod that don't set the required attribute in the generated
html.


code:
One problem with the current configurators for remotes is they have a lot
of notes and text to read at the top. I've worked on cut that down somewhat,
mostly by adding links and help buttons next to fields in the form.

I also made each form have a check box controlling whether encryption is
enabled. Mostly as a way to declutter the text at the top, which always had
to say encryption is enabled.

---

I have a fairly well worked out design for [[desymlink]]. Will wait a few
days to work on it to let it jell.


code:
Biding my time while [[desymlink]] gells in my head..

Fixed a bug in the assistant's local pairing that rejected ssh keys with a
period in the comment.

Fixed a bug in the assistant that made it try to drop content from remotes
that didn't have it, and avoided a drop failure crashing a whole assistant
thread.

Made --auto behave better when preferred content is set.

Looked into making the transfer queue allow running multiple transfers at
the same time, ie, one per remote. This seems to essentially mean splitting
the queue into per remote queues. There are some complexities, and I
decided not to dive into working through it right now, since it'd be
a distraction from thinking about [[desymlink]]. Will revisit it later.

Allow specifying a port when setting up a ssh remote.

While doing that, noticed that the assistant fails to transfer files to
sync to a ssh remote that was just added. That got broken while optimising
reconnecting with a remote; fixed it.


code:
Started laying the groundwork for [[desymlink]]'s direct mode.
I got rather far!

A git repo can be configured with `annex.direct` and all actions that
transfer objects to it will replace the symlinks with regular files.
Removing objects also works (and puts back a broken symlink),
as does checking if an object is present, which even detects if a file
has been modified.

So far, this works best when such a direct mode repository is used as a git
remote of another repository. It is also possible to run a few git-annex
commands, like "get" in a direct mode repository, though others, like
"drop" won't work because they rely on the symlink to map back to the key.

Direct mode relies on map files existing for each key in the repository, that tell
what file(s) use it. It also relies on cache files, that contain the last
known mtime, size, and inode of the file. So far, I've been setting these
files up by hand.

The main thing that's missing is support for transferring objects from
direct mode repositories. There's no single place I can modify to support
that (like I could for the stuff mentioned above), and also it's difficult
to do safely, since files could be modified at any time.

So it'll need to quarantine files, to prevent a modified version from
getting sent out. I could either do this by copying the file, or by
temporarily `git annex lock`ing it. Unsure which choice would be less
annoying..

----

Also did some investigation with Jimmy of the OSX app git-config hang.
Seems to be some kind of imcompatability between the 10.7 autobuilder and
10.8. Next step is probably to try to build on 10.8. Might also
be worth trying <http://macdylibbundler.sourceforge.net/>, although my
own scripts do more or less the same thing to build the app.


code:
[[!comment format=mdwn
 username="http://lj.rossia.org/users/imz/"
 ip="79.165.59.119"
 subject=""removing"  vs drop"
 date="2012-12-12T13:20:42Z"
 content="""
I don't understand the difference behind:

> Removing objects also works (and puts back a broken symlink)

and

> \"drop\" won't work because they rely on the symlink to map back to the key.

If a file is removed (its content, which is replaced by a symlink), then it's not present there, so effectively it should be counted as \"dropped\" at this place. So, removing a file without counting it as dropped is something inconsistent, isn't it? Do I misunderstand something?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.8.117"
 subject="comment 2"
 date="2012-12-12T23:45:42Z"
 content="""
`git annex drop` is a user-level operation built on top of lower-level object removal functions that are also used by other things.
"""]]


code:
Got object sending working in direct mode. However, I don't yet have a
reliable way to deal with files being modified while they're being
transferred. I have code that detects it on the sending side, but the
receiver is still free to move the wrong content into its annex, and record
that it has the content. So that's not acceptable, and I'll need to work
on it some more. However, at this point I can use a direct mode repository
as a remote and transfer files from and to it.

----

Automated updating of the cached mtime, etc data. Next I need to automate
generation of the key to filename mapping files. I'm thinking that I'll make
`git annex sync` do it. Then, once I get committing and
merging working in direct mode repositories (which is likely to be a
good week's worth of work), the workflow for using these repositories
will be something like this:

	git config annex.direct true
	git annex sync # pulls any changes, merges, updates maps and caches
	git annex get
	# modify files
	git annex sync # commits and pushes changes

And once I get direct mode repositories working to this degree at the
command line, I can get on with adding support to the assistant.

----

Also did some more work today on the OSX app. Am in the middle of getting
it to modify the binaries in the app to change the paths to the libraries they
depend on. This will avoid the hacky environment variable it is currently
using, and make runshell a much more usable environment. It's the right way
to do it. (I can't believe I just said RPATH was the right way to do
anything.)

In the middle of this, I discovered
<http://hackage.haskell.org/package/cabal-macosx>, which does the same
type of thing.

Anyway, I have to do some crazy hacks to work around short library name
fields in executables that I don't want to have to be specially rebuilt in
order to build the webapp. Like git.


code:
Made `git annex sync` update the file mappings in direct mode.
To do this efficiently, it uses `git diff-tree` to find files that are
changed by the sync, and only updates those mappings. I'm rather happy
with this, as a first step to fully supporting sync in direct mode.

Finished the overhaul of the OSX app's library handling. It seems to work
well, and will fix a whole class of ways the OSX app could break.

Fixed a bug in the preferred content settings for backup repositories,
introduced by some changes I made to preferred content handling 4 days ago.

Fixed the Debian package to build with WebDAV support, which I forgot to
turn on before.

Planning a release tomorrow.


code:
... I'm getting tired of kqueue.

But the end of the tunnel is in sight. Today I made git-annex handle files
that are still open for write after a kqueue creation event is received.
Unlike with inotify, which has a new event each time a file is closed,
kqueue only gets one event when a file is first created, and so git-annex
needs to retry adding files until there are no writers left.

Eventually I found an elegant way to do that. The committer thread already
wakes up every second as long as there's a pending change to commit. So for
adds that need to be retried, it can just push them back onto the change
queue, and the committer thread will wait one second and retry the add. One
second might be too frequent to check, but it will do for now.

This means that `git annex watch` should now be usable on OSX, FreeBSD, and
NetBSD! (It'll also work on Debian kFreeBSD once [lsof is ported to it](http://bugs.debian.org/589103).)
I've meged kqueue support to `master`.

I also think I've squashed the empty commits that were sometimes made.

Incidentally, I'm 50% through my first month, and finishing [[inotify]]
was the first half of my roadmap for this month. Seem to be right on
schedule.. Now I need to start thinking about [[syncing]].


code:
Pondering [[syncing]] today. I will be doing syncing of the git repository
first, and working on syncing of file data later.

The former seems straightforward enough, since we just want to push all
changes to everywhere. Indeed, git-annex already has a [[sync]] command
that uses a smart technique to allow syncing between clones without a
central bare repository. (Props to Joachim Breitner for that.)

But it's not all easy. Syncing should happen as fast as possible, so
changes show up without delay. Eventually it'll need to support syncing
between nodes that cannot directly contact one-another. Syncing needs to
deal with nodes coming and going; one example of that is a USB drive being
plugged in, which should immediately be synced, but network can also come
and go, so it should periodically retry nodes it failed to sync with. To
start with, I'll be focusing on fast syncing between directly connected
nodes, but I have to keep this wider problem space in mind.

One problem with `git annex sync` is that it has to be run in both clones
in order for changes to fully propagate. This is because git doesn't allow
pushing changes into a non-bare repository; so instead it drops off a new
branch in `.git/refs/remotes/$foo/synced/master`. Then when it's run locally
it merges that new branch into `master`.

So, how to trigger a clone to run `git annex sync` when syncing to it?
Well, I just realized I have spent two weeks developing something that can
be repurposed to do that! [[Inotify]] can watch for changes to
`.git/refs/remotes`, and the instant a change is made, the local sync
process can be started. This avoids needing to make another ssh connection
to trigger the sync, so is faster and allows the data to be transferred
over another protocol than ssh, which may come in handy later.

So, in summary, here's what will happen when a new file is created:

1. inotify event causes the file to be added to the annex, and
   immediately committed.
2. new branch is pushed to remotes (probably in parallel)
3. remotes notice new sync branch and merge it
4. (data sync, TBD later)
5. file is fully synced and available

Steps 1, 2, and 3 should all be able to be accomplished in under a second.
The speed of `git push` making a ssh connection will be the main limit
to making it fast. (Perhaps I should also reuse git-annex's existing ssh
connection caching code?)


code:
Yesterday I cut another release. However, getting an OSX build took until
12:12 pm today because of a confusion about the location of lsof on OSX. The
OSX build is now available, and I'm looking forward to hearing if it's working!

----

Today I've been working on making `git annex sync` commit in direct mode.

For this I needed to find all new, modified, and deleted files, and I also
need the git SHA from the index for all non-new files. There's not really
an ideal git command to use to query this. For now I'm using
`git ls-files --others --stage`, which works but lists more files than I
really need to look at. It might be worth using one of the Haskell libraries
that can directly read git's index.. but for now I'll stick with `ls-files`.

It has to check all direct mode files whose content is present, which means
one stat per file (on top of the stat that git already does), as well as one
retrieval of the key per file (using the single `git cat-file` process that
git-annex talks to).

This is about as efficient as I can make it, except that unmodified
annexed files whose content is not present are listed due to --stage,
and so it has to stat those too, and currently also feeds them into `git add`.

The assistant will be able to avoid all this work, except once at startup.

Anyway, direct mode committing is working!

For now, `git annex sync` in direct mode also adds new files. This because
`git annex add` doesn't work yet in direct mode.

It's possible for a direct mode file to be changed during a commit,
which would be a problem since committing involves things like calculating
the key and caching the mtime/etc, that would be screwed up. I took
care to handle that case; it checks the mtime/etc cache before and after
generating a key for the file, and if it detects the file has changed,
avoids committing anything. It could retry, but if the file is a VM disk
image or something else that's constantly modified, commit retrying forever
would not be good.

----

For `git annex sync` to be usable in direct mode, it still needs
to handle merging. It looks like I may be able to just enhance the automatic
conflict resolution code to know about typechanged direct mode files.

The other missing piece before this can really be used is that currently
the key to file mapping is only maintained for files added locally, or
that come in via `git annex sync`. Something needs to set up that mapping
for files present when the repo is initally cloned. Maybe the thing
to do is to have a `git annex directmode` command that enables/disables
direct mode and can setup the the mapping, as well as any necessary unlocks
and setting the trust level to untrusted.


code:
Built `git annex direct` and `git annex indirect` to toggle back and forth
between direct mode. Made `git annex status` show if the repository is in
direct mode. Now *only* merging is needed for direct mode to be basically
usable.

I can do a little demo now. Pay attention to the "@" ls shows at the end
of symlinks.

	joey@gnu:~/tmp/bench/rdirect>ls
	myfile@  otherfile@
	joey@gnu:~/tmp/bench/rdirect>git annex find
	otherfile
	# So, two files, only one present in this repo.

	joey@gnu:~/tmp/bench/rdirect>git annex direct
	commit
	# On branch master
	# Your branch is ahead of 'origin/master' by 7 commits.
	#
	nothing to commit (working directory clean)
	ok
	direct myfile ok
	direct otherfile ok
	direct  ok

	joey@gnu:~/tmp/bench/rdirect>ls
	myfile@  otherfile
	# myfile is still a broken symlink because we don't have its content
	joey@gnu:~/tmp/bench/rdirect>git annex get myfile
	get myfile (from origin...) ok
	(Recording state in git...)
	joey@gnu:~/tmp/bench/rdirect>ls
	myfile  otherfile

	joey@gnu:~/tmp/bench/rdirect>echo "look mom, no symlinks" >> myfile
	joey@gnu:~/tmp/bench/rdirect>git annex sync
	add myfile (checksum...) ok
	commit
	(Recording state in git...)
	[master 0e8de9b] git-annex automatic sync
	...
	ok

	joey@gnu:~/tmp/bench/rdirect>git annex indirect
	commit  ok
	indirect myfile ok
	indirect otherfile ok
	indirect  ok
	joey@gnu:~/tmp/bench/rdirect>ls
	myfile@  otherfile@

I'd like `git annex direct` to set the repository to untrusted, but
I didn't do it. Partly because having `git annex indirect` set it back to
semitrusted seems possibly wrong -- the user might not trust a repo even in
indirect mode. Or might fully trust it. The docs will encourage users to
set direct mode repos to untrusted -- in direct mode you're operating
without large swathes of git-annex's carefully constructed safety net.
(When the assistant later uses direct mode, it'll untrust the repository
automatically.)


code:
Fixed a bug in the kqueue code that made the assistant not notice when a
file was renamed into a subdirectory. This turned out to be because the
symlink got broken, and it was using `stat` on the file. Switching to
`lstat` fixed that.

Improved installation of programs into standalone bundles. Now it uses
the programs detected by configure, rather than a separate hardcoded list.
Also improved handling of lsof, which is not always in PATH.

Made a OSX 10.8.2 build of the app, which is nearly my last gasp attempt
at finding a way around this crazy `git init` spinning problem with Jimmy's
daily builds are used with newer OSX versions. Try it here:
<http://downloads.kitenet.net/tmp/git-annex.dmg.bz2>

----

Mailed out the Kickstarter T-shirt rewards today, to people in the US.
Have to fill out a bunch of forms before I can mail the non-US ones.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmYiJgOvC4IDYkr2KIjMlfVD9r_1Sij_jY"
 nickname="Douglas"
 subject="OSX troubles"
 date="2012-12-17T16:07:41Z"
 content="""
Have you considered making a homebrew installable version of git-annex? It may make things easier for you when it comes to build dependencies.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.8.117"
 subject="comment 2"
 date="2012-12-17T16:16:47Z"
 content="""
I would *love* *love* *love* for there to be a homebrew for git-annex. But I know nothing about homebrew so am not in a very good position to do it.
"""]]


code:
As winter clouds set in, I have to ration my solar power and have been less
active than usual.

It seems that the OSX 10.8.2 `git init` hanging issue has indeed been
resolved, by building the app on 10.8.2. Very good news!  Autobuilder setup is
in progress.

----

Finally getting stuck in to direct mode git-merge handling. It's
not possible to run `git merge` in a direct mode tree, because it'll
see typechanged files and refuse to do anything.

So the only way to use `git merge`, rather than writing my own merge engine,
is to use `--work-tree` to make it operate in a temporary work tree directory
rather than the real one.

When it's run this way, any new, modified, or renamed files will be added
to the temp dir, and will need to be moved to the real work tree.
To detect deleted files, need to use `git ls-files --others`, and
look at the old tree to see if the listed files were in it.

When a merge conflict occurs, the new version of the file will be in the temp
directory, and the old one in the work tree. The normal automatic merge
conflict resolution machinery should work, with just some tweaks to handle
direct mode.


code:
Got merging working in direct mode!

Basically works as outlined yesterday, although slightly less clumsily.
Since there was already code that ran `git diff-tree` to update the
associated files mappings after a merge, I was able to adapt that same code
to also update the working tree.

An important invariant for direct mode merges is that they should never
cause annexed objects to be dropped. So if a file is deleted by a merge,
and was a direct mode file that was the only place in the working copy
where an object was stored, the object is moved into `.git/annex/objects`.
This avoids data loss and any need to re-transfer objects after a merge.
It also makes renames and other move complex tree manipulations always end
up with direct mode files, when their content was present.

Automatic merge conflict resoltion doesn't quite work right yet in direct
mode.

Direct mode has landed in the `master` branch, but I still consider it
experimental, and of course the assistant still needs to be updated to
support it.


code:
Finished getting automatic merge conflict resolution working in direct
mode. Turned out I was almost there yesterday, just a bug in a filename
comparison needed to be fixed.

Fixed a bug where the assistant dropped a file after transferring it,
despite the preferred content settings saying it should keep its copy of
the file. This turned out to be due to it reading the transfer info
incorrectly, and adding a "\n" to the end of the filename, which caused the
preferred content check to think it wasn't wanted after all. (Probably
because it thought 0 copies of the file were wanted, but I didn't look into
this in detail.)

Worked on my test suite, particularly more QuickCheck tests. I need to
use QuickCheck more, particularly when I've pairs of functions, like encode
and decode, that make for easy QuickCheck properties.


code:
Over Christmas, I'm working on making the assistant support direct
mode. I like to have a fairly detailed plan before starting this kind of
job, but in this case, I don't. (Also I have a cold so planning? Meh.)
This is a process of seeing what's broken in direct mode and fixing it.
I don't know if it'll be easy or hard. Let's find out..

* First, got direct mode adding of new files working. This was not hard, all the
  pieces I needed were there. For now, it uses the same method as in
  indirect mode to make sure nothing can modify the file while it's being
  added.

* An unexpected problem is that in its startup scan, the assistant runs
  `git add --update` to detect and stage any deletions that happened
  while it was not running. But in direct mode that also stages the full file
  contents, so can't be used. Had to switch to using git plumbing to only
  stage deleted files. Happily this also led to fixing a bug; deletions
  were not always committed at startup when using the old method; with the
  new method it can tell when there are deletions and trigger a commit.

* Next, got it to commit when direct mode files are modified. The Watcher
  thread gets a inotify event when this happens, so that was easy. (Although
  in doing that I had to disable a guard in direct mode that made annexed
  files co-exist with regular in-git files, so such mixed repositories
  probably won't work in direct mode yet.)

  However, naughty kqueue is another story, there are no kqueue events for
  file modifications. So this won't work on OSX or the BSDs yet. I tried
  setting some more kqueue flags in hope that one would make such events
  appear, but no luck. Seems I will need to find some other method to detect
  file modifications, possibly an OSX-specific API.

* Another unexpected problem: When an assistant receives new files from one
  of its remotes, in direct mode it still sets up symlinks to the content.
  This was because the Merger thread didn't use the `sync` command's direct
  mode aware merge code.. so fixed that.

* Finally there was some direct mode bookeeping the assistant has
  to get right. For example, when a file is modified, the old object has
  to be looked up, and be marked as not locally present any longer. That
  lookup relies on the already running `git cat-file --batch`, so it's
  not as fast as it could be, if I kept a local cache of the mapping
  between files and objects. But it seems fast enough for now.

At this point the assistant seems to work in direct mode on Linux!
Needs more testing..


code:
Investigated using the OSX fsevents API to detect when files are modified,
so they can be committed when using direct mode. There's a
[haskell library](http://hackage.haskell.org/package/hfsevents-0.1.3)
and even a [sample directory watching program](http://hackage.haskell.org/package/hobbes).
Initial tests look good...

Using fsevents will avoid kqueue's problems with needing enough file
descriptors to open every subdirectory. kqueue is a rather poor match for
git-annex's needs, really. It does not seem to provide events for file
modifications at all, unless every *file* is individually opened. While I
dislike leaving the BSD's out, they need a better interface to be perfectly
supported by git-annex, and kqueue will still work for indirect mode
repositories.

----

Got the assistant to use fsevents. It seems to work well!

The only problem I know of is that it doesn't yet handle whole directory
renames. That should be easy to fix later.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkey8WuXUh_x5JC2c9_it1CYRnVTgdGu1M"
 nickname="Dustin"
 subject="Linux Support"
 date="2012-12-27T21:37:45Z"
 content="""
Does this mean Linux will not support Direct mode?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.127"
 subject="comment 2"
 date="2012-12-28T02:46:23Z"
 content="""
No, linux's inotify has everything I need for direct mode.
"""]]


code:
Short day today, but I spent it all on testing the new FSEvents code,
getting it working with the assistant in direct mode. This included fixing
its handling of renaming, and various other bugs.

The assistant in direct mode now seems to work well on OSX. So I made
the assistant *default* to making direct mode repositories on OSX.

That'll presumably flush out any bugs. :) More importantly,
it let me close several OSX-specific bugs to do with interactions between
git-annex's symlinks and OSX programs that were apparently written under the
misprehension that it's a user-mode program's job to manually follow symlinks.

Of course, defaulting to direct mode also means users can just modify files
as they like and the assistant will commit and sync the changed files.
I'm waiting to see if direct mode becomes popular enough to make it the
default on all OS's.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm5iosFbL2By7UFeViqkc6v-hoAtqILeDA"
 nickname="Laszlo"
 subject="comment 1"
 date="2012-12-28T23:53:22Z"
 content="""
Make a release please, Im eager to try it out!:)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnFhHPFP7j3wGNgBxEJoA8LcBJ4Xd1tTMY"
 nickname="Joe"
 subject="Release + Version Number"
 date="2012-12-29T11:16:05Z"
 content="""
I second Laszlo's request for a release! Also, could you include a version number of some sort on the download (and binary if there isn't one there) to make it easy to tell what is available for download vs. what is installed?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 3"
 date="2012-12-29T13:26:30Z"
 content="""
There are daily/hourly builds at the [[install/OSX]] install page, try out the new features from there
"""]]


code:
Syncing works! I have two clones, and any file I create in the first
is immediately visible in the second. Delete that file from the second, and
it's immediately removed from the first.

Most of my work today felt like stitching existing limbs onto a pre-existing
monster. Took the committer thread, that waits for changes and commits them,
and refashioned it into a pusher thread, that waits for commits and pushes
them. Took the watcher thread, that watches for files being made,
and refashioned it into a merger thread, that watches for git refs being
updated. Pulled in bits of the `git annex sync` command to reanimate this.

It may be a shambling hulk, but it works.

Actually, it's not much of a shambling hulk; I refactored my code after
copying it. ;)

I think I'm up to 11 threads now in the new
`git annex assistant` command, each with its own job, and each needing
to avoid stepping on the other's toes. I did see one MVar deadlock error
today, which I have not managed to reproduce after some changes. I think
the committer thread was triggering the merger thread, which probably
then waited on the Annex state MVar the committer thread had held.

Anyway, it even pushes to remotes in parallel, and keeps track of remotes
it failed to push to, although as of yet it doesn't do any attempt at
periodically retrying.

One bug I need to deal with is that the push code assumes any change
made to the remote has already been pushed back to it. When it hasn't,
the push will fail due to not being a fast-forward. I need to make it
detect this case and pull before pushing.

(I've pushed this work out in a new `assistant branch`.)


code:
A few final bits and pieces of direct mode. Fixed a few more bugs in the
assistant. Made all git-annex commands that don't work at
all, or only partially work in direct mode, refuse to run at all. Also,
some optimisations.

I'll surely need to revisit direct mode later and make more commands
support it; `fsck` and `add` especially.
But the only thing I'd like to deal with before I make a release with direct
mode is the problem of files being able to be modified while they're
being transferred, which can result in data loss.


code:
Released the first git-annex with direct mode today. Notably, the assistant
enables direct mode in repositories it creates. All builds are updated to
3.20130102 now.

My plan for this month is to fix whatever things currently might be
preventing you from using the git-annex assistant. So bugfixes and
whatever other important gaps need to be filled, but no major new
feature developments.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm5iosFbL2By7UFeViqkc6v-hoAtqILeDA"
 nickname="Laszlo"
 subject="comment 1"
 date="2013-01-02T21:26:32Z"
 content="""
1. Im not able to set direct mode (linux build, Version: 3.20130102 in about page).
Is there something specific needed? I started fresh. I added a new directory (empty), once
git annex assistant started, I copied files to that directory, and the assistant
replaced all files with symlinks, just as before.
What am I missing here?

2. It is just minor usability remark:
 I would like to see in the gui somewhere shown where the actual config is stored (~/.config/git-annex),
and an option to wipe it out and start fresh. Also a restart button.

Best, Laszlo

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.211"
 subject="comment 2"
 date="2013-01-03T17:59:22Z"
 content="""
\"I added a new directory (empty)\"

Sounds like you set up the repository yourself, rather than letting the assistant do it. If so, it won't be in direct mode. You can `git annex assistant--stop`, and then `git annex direct` to enable direct mode.

Restart/shutdown in the UI coming soon, I think.

The entire git-annex configuration, include that of the assistant, is stored in the `.git/config` of your reposository, and on its `git-annex` branch. Deleting the repository is thus all that's needed to start over.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm5iosFbL2By7UFeViqkc6v-hoAtqILeDA"
 nickname="Laszlo"
 subject="comment 3"
 date="2013-01-03T22:23:36Z"
 content="""
>> I added a new directory (empty)

> Sounds like you set up the repository yourself, rather than letting the assistant do it.
> If so, it won't be in direct mode. You can git annex assistant--stop,
> and then git annex direct to enable direct mode.

I added like this:
0. no git-annex or git-annex-assistant is running
1. create directory (/mnt/dxd/annex)
2. launch git-annex-assistant
3. add repository via the webbrowser
4. copy a file over this(/mnt/dxd/annex) directory (via command line or nautilus)

The file is replaced with symlink.

This fix the issue:
0. stop git-annex-assistant
1. Go in the shell to that directory (cd /mnt/dxd/annex)
2. switch to direct mode (/home/looser/Desktop/download/git-annex.linux/git-annex direct)
3. start git-annex-assistant
4. create a new repository(/mnt/dxd/annex2) via the webbrowser (add repository in top right corner)
5. Now:
/mnt/dxd/annex - NOT in direct mode (symlinks created instantly)
/mnt/dxd/annex2 - in direct mode

Im really happy I have finally direct mode running.
I will surely stress-test git-annex in the coming weeks.

> Restart/shutdown in the UI coming soon, I think.

Awesome.

> The entire git-annex configuration, include that of the assistant,
> is stored in the .git/config of your reposository, and on its git-annex branch.
> Deleting the repository is thus all that's needed to start over.

Thank you for the answer.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm5iosFbL2By7UFeViqkc6v-hoAtqILeDA"
 nickname="Laszlo"
 subject="comment 4"
 date="2013-01-03T22:33:54Z"
 content="""
Usecase:
Keep two directories in sync on the same computer.

\"Removable drive\" almost covers this, except you need an usb thumbdrive
permanently plugged into the computer.

Would be nice, if I could select a \"sync to this local folder\" option
alongside the \"Select drive:\" option.

So it is my choice if I want to sync to a partition or to a folder.

Bug(?):
I created two repositories:
/mnt/dxd/annex
/mnt/dxd/annex2

And I added a \"Removable drive\" syncing option to /mnt/dxd/annex2, and I selected the \"/mnt/dxd\" drive as removable drive.
The two repositories got mixed together
(ie. broken symlinks pointing to nonexistent .git/annex/objects/.. file)

Best,
 Laszlo

"""]]


code:
[[!comment format=c
 username="https://www.google.com/accounts/o8/id?id=AItOawm5iosFbL2By7UFeViqkc6v-hoAtqILeDA"
 nickname="Laszlo"
 subject="comment 5"
 date="2013-01-03T22:49:41Z"
 content="""
Ok, I think I figured it out. I have quit from git-annex-assistant (Ctrl-C in terminal window).

And launched this command:
$ ps -e |grep git-ann
 1782 ?        00:01:00 git-annex
20144 pts/0    00:00:00 git-annex-webap
20148 pts/0    00:00:00 git-annex
20178 pts/0    00:00:00 git-annex <defunct>
20508 pts/0    00:00:00 git-annex <defunct>
20528 ?        00:00:02 git-annex
21132 ?        00:00:00 git-annex <defunct>

I will be more cautious in future.
"""]]


code:
[[!comment format=c
 username="https://www.google.com/accounts/o8/id?id=AItOawm5iosFbL2By7UFeViqkc6v-hoAtqILeDA"
 nickname="Laszlo"
 subject="comment 6"
 date="2013-01-03T22:55:41Z"
 content="""
another problem:
1. If you add a removable drive via the gui, you can not remove it anymore (from the gui)
2. If you delete the directory on the removable drive, git-annex assistant silently fails
to recreate the directory and start fresh.
2/a. on the gui, there is no indication something is wrong with the removable drive repository
2/b on the terminal it shows only this:
fatal: Not a git repository: '/mnt/dxd/annex/.git'

On the main page it shows the two repositories, but on the upper right corner, the dropdown menu
contains only the original repository and not the removeable drive repository.

I stop spaming now...

Laszlo

"""]]


code:
[[!comment format=c
 username="https://www.google.com/accounts/o8/id?id=AItOawm5iosFbL2By7UFeViqkc6v-hoAtqILeDA"
 nickname="Laszlo"
 subject="comment 7"
 date="2013-01-03T23:07:27Z"
 content="""
Can not withstand myself.

1. If you add a Removable drive repository, only the *new* files are copied over to the removable drive.
2. Removable drive repository is not in direct mode.
Would be nice if it could be kept the same mode as the original repository.
It is some kind of special repsoitory, I do not see any seemlinks either.
"""]]


code:
[[!comment format=mdwn
 username="http://csights.myopenid.com/"
 ip="70.226.161.163"
 subject="walkthroughs?"
 date="2013-01-04T02:39:59Z"
 content="""
Could you post some walk-throughs?
Is it possible to set up a direct mode repository using git-annex webapp ?

Thanks for your work!  Sad to hear you had to get a generator.  :)
C.

"""]]


code:
[[!comment format=mdwn
 username="spwhitton"
 ip="163.1.166.255"
 subject="comment 9"
 date="2013-01-04T20:26:46Z"
 content="""
You noted in a recent blog post that direct mode won't work with mixed repositories, that is, those that have files checked into git alongside annexed ones.  Is this still the case?  Thanks.
"""]]


code:
[Installed a generator](http://joeyh.name/blog/entry/overcast/),
so I'll have more power and less hibernation.

Added UI in the webapp to shut down the daemon. Would like to also have
restart UI, but that's rather harder to do, seems it'd need to start
another copy of the webapp (and so, of the assistant), and redirect the
browser to its new url. ... But running two assistants in the same repo at
the same time isn't good. Anyway, users can now use the UI to shut it down,
and then use their native desktop UI to start it back up.

[[!img /assistant/controlmenu.png]]

Spiffed up the control menu. Had to stop listing other local repositories
in the menu, because there was no way to notice when a new one was added
(without checking a file on every page load, which is way overkill for this
minor feature). Instead added a new page that lists local repositories it
can switch to.


code:
There was a typo in cabal file that broke building the assistant on OSX.
This didn't affect the autobuilds of the app, but several users building by
hand reported problems. I made a new minor release fixing that typo, and
also a resouce leak bug.

Got a restart UI working after all. It's a hack though. It
opens a new tab for the new assistant instance, and as most web browsers
don't allow javascript to close tabs, the old tab is left open. At some
point I need to add a proper thread manager to the assistant, which the
restart code could use to kill the watcher and committer threads, and then
I could do a clean restart, bringing up the new daemon and redirecting the
browser to it.

Found a bug in the assistant in direct mode -- the expensive transfer scan
didn't queue uploads needed to sync to other repos in direct mode, although
it did queue downloads. Fixing this laid some very useful groundwork for
making more commands support direct mode, too. Got stuck for a long time
dealing with some very strange `git-cat-file` behavior while making this
work. Ended up putting in a workaround.

After that, I found that these commands work in direct mode, without
needing any futher changes!

* `git annex find`
* `git annex whereis`
* `git annex copy`
* `git annex move`
* `git annex drop`
* `git annex log`

Enjoy! The only commands I'd like to add to this are `fsck`, `add`,
and `addurl`...


code:
Several bugfixes from user feedback today.

Made the assistant detect misconfigured systems where git will fail to
commit because it cannot determine the user's name or email address, and
dummy up enough info to get git working. It makes sense for git and
git-annex to fail at the command line on such a misconfigured system, so
the user can fix it, but for the assistant it makes sense to plow on and just
work.

I found a big gap in direct mode -- all the special remotes expected to find
content in the indirect mode location when transferring to the remote. It
was easy to fix once I noticed the problem. This is a big enough bug that
I will be making a new release in a day or so.

Also, got fsck working in direct mode. It doesn't check as many things
as in indirect mode, because direct mode files can be modified at any time.
Still, it's usable, and useful.


code:
(Posted a day late.) [[!meta date="Mon Jan  7 16:05:13 JEST 2013"]]

Got `git annex add` (and `addurl`) working in direct mode. This allowed me
to make `git annex sync` in direct mode no longer automatically add new
files.

It's also now safe to mix direct mode annexed files with regular files in
git, in the same repository. Might have been safe all along, but I've
tested it, and it certainly works now. You just have to be careful to not
use `git commit -a` to commit changes to such files, since that'll also
stage the entire content of direct mode files.

Made a minor release for these recent changes and bugfixes. Recommended if
you're using direct mode. Had to chase down a stupid typo I made yesterday
that caused fsck to infinite loop if it found a corrupted file. Thank
goodness for test suites.


code:
I was up at the crack of dawn wrestling 100 pound batteries around for 3
hours and rewiring most of my battery bank, so today is a partial day...
but a day with power, which is always nice.

Did some design work on finally making transfers of files from direct mode
repositories safe, even if a file is modified as it's being uploaded.
This seems easily doable for special remotes; git to git repository
transfers are harder, but I think I see how to do it without breaking
backwards compatability.

(An unresolved problem is that a temp file would be left behind when a
transfer failed due to a file being changed. What would really be nice to
do is to use that temp file as the rsync basis when transferring the new
version of a file. Although this really goes beyond direct mode, and into
[[deltas]] territory.)

Made fsck work better in direct mode repositories. While it's expected for
files to change at any time in direct mode, and so fsck cannot complain
every time there's a checksum mismatch, it is possible for it to detect
when a file does not *seem* to have changed, then check its checksum,
and so detect disk corruption or other data problems.

Also dealt with several bug reports. One really weird one involves `git
cat-file` failing due to some kind of gpg signed data in the git-annex
branch. I don't understand that at all yet.


code:
Well underway on making direct mode transfers roll back when the file is
modified while it's transferred.

As expected, it was easy to do for all the special remotes ... Except for
bup, which does not allow deleting content. For bup it just removes the git
ref for the bad content, and relies on bup's use of git delta compression
to keep space use sane.

The problem is also handled by `git-annex-shell sendkey`.
But not yet for downloads from other git repositories. Bit stuck on that.

Also: A few minor bug fixes.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm7AuSfii_tCkLyspL6Mr0ATlO6OxLNYOo"
 nickname="Georg"
 subject="comment 1"
 date="2013-01-10T08:49:20Z"
 content="""
hmm, wouldn't it make sense to do the downloads into the annex object space, just like with indirect annexes? Then when the download is complete, with indirect annexes you create a soft link and with direct annexes you move the file (or create a hardlink and remove the directory entry in the annex object space)? Actually when you started to talk about direct I first envisioned it to be exactly like indirect, only with hard links instead of soft links.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.7.238"
 subject="comment 2"
 date="2013-01-10T16:06:28Z"
 content="""
That is how direct mode works. (Although without the hard links, since maintaining them doesn't have any benefits.) And that's how downloading into a direct mode repo works. The issue I'm dealing with, though, is uploading from a direct mode repo, where the file that's being uploaded can be modified at the same time.
"""]]


code:
This month's theme is supposed to be fixing up whatever might prevent users
from using the assistant. To that end, I've posted an open-ended poll,
[[polls/what is preventing me from using git-annex assistant]]. Please go
fill it out so I can get an idea of how many people are using the
assistant, and what needs to be done to get the rest of you, and your
friends and family able to use it.

In the meantime, today I fixed several bugs that were recently reported in
the webapp and assistant. Getting it working as widely as possible, even on
strange IPv6 only systems, and with browsers that didn't like my generated
javascript code is important, and fits right into this month's theme. I'm
happy to see lots of bugs being filed, since it means more users are trying
the assistant out.

Also continued work on making direct mode transfers safe. All transfers to
local git remotes (wish I had a better phrase!) are now safe in direct mode.
Only uploading from a direct mode repository over ssh to another git
repository is still potentially unsafe.


code:
[[!comment format=mdwn
 username="http://jasonwoof.com/"
 nickname="JasonWoof"
 subject="phrase suggestion"
 date="2013-01-11T00:24:39Z"
 content="""
I suggest using the term \"repository\" not \"remote\". Two reasons: 1: it's a noun in *English. 2: it works for both local and remote repositories.

Both get +1 because they match git terminology. (I have a secret dream that the assistant might bring people to git, maybe that's silly.)

\* OK, so \"remote\" is also a noun in English, but it means the widget with buttons that controls the tv/whatever.
"""]]


code:
[[!comment format=mdwn
 username="http://csights.myopenid.com/"
 ip="70.226.161.163"
 subject=""local git remotes""
 date="2013-01-13T02:36:48Z"
 content="""
I am not quite sure how git-annex works and I do not know what a \"local git remote\" functionality is.

How about this: call a \"git remote\" a \"git-annex store\" or just \"store\".  Then instead of a remote git remote it would be a remote git-annex store or just remote store.  :)

"""]]


code:
I've finished making direct mode file transfers safe. The last piece of the
puzzle was making `git-annex-shell recv-key` check data it's received from
direct mode repositories. This is a bit expensive, but avoids adding
another round-trip to the protocol. I might revisit this later, this was
just a quick fix.

---

The [[poll|polls/what is preventing me from using git-annex assistant]] was
quite useful. Some interesting points:

* 14% have been reading this blog, and rightly don't trust direct mode to
  be safe. Which is why I went ahead with a quick fix to make it safe.
* 6% want an Ubuntu PPA. I don't anticipate doing this myself, but
  if anyone who develops for Ubuntu wants to put together a PPA with a
  newer version, I can help you pick the newer haskell packages you'll
  need from Debian, etc.
* 9% just need me to update the amd64 build in Debian sid. I forgot to
  include it in the last release, and the Debian buildds cannot currently
  autobuild git-annex due to some breakage in the versions of haskell
  libraries in unstable. Hopefully I'll remember to include an amd64 build
  in my next release.

And lots of other interesting stuff, I have a nice new TODO list now. :)


code:
[[!comment format=mdwn
 username="spwhitton"
 ip="163.1.166.255"
 subject="comment 1"
 date="2013-01-11T20:25:48Z"
 content="""
If you have time, an updated sid package for armel would be much appreciated.
"""]]


code:
[[!comment format=mdwn
 username="http://csights.myopenid.com/"
 ip="70.226.161.163"
 subject="use debian repos instead of Ubuntu PPA?"
 date="2013-01-12T13:43:44Z"
 content="""
Maybe Ubuntu users would be satisfied with tweaks to their sources.list and preferences?

/etc/apt/sources.list.d/git-annex.sources.list
deb http://ftp.us.debian.org/debian unstable main

/etc/apt/preferences.d/git-annex.preferences
Package: *
Pin: release a=unstable,o=Debian
Pin-Priority: 400

Not sure how that will work out in Ubuntu package manager.  In aptitude you'll get warning about untrusted repos (maybe install Debian's gpg keys) and will probably have to manually select dependencies from Debian repos.
"""]]


code:
I released a version of git-annex over the weekend that includes the `git
annex watch` command. There's a minor issue installing it from cabal on
OSX, which I've fixed in my tree. Nice timing: At least the watch command
should be shipped in the next Debian release, which freezes at the end of
the month.

Jimmy found out how kqueue [[blows
up|bugs/Issue_on_OSX_with_some_system_limits]] when there are too many
directories to keep all open. I'm not surprised this happens, but it's nice
to see exactly how. Odd that it happened to him at just 512 directories;
I'd have guessed more. I have plans to fork watcher programs that each
watch 512 directories (or whatever the ulimit is), to deal with this. What
a pitiful interface is kqueue.. I have not thought yet about how the watcher
programs would communicate back to the main program.

----

Back on the assistant front, I've worked today on making git syncing more
robust. Now when a push fails, it tries a pull, and a merge, and repushes.
That ensures that the push is, almost always, a fast-forward. Unless
something else gets in a push first, anyway!

If a push still fails, there's Yet Another Thread, added today, that will
wake up after 30 minutes and retry the push. It currently keeps retrying
every 30 minutes until the push finally gets though. This will deal, to
some degree, with those situations where a remote is only sometimes
available.

I need to refine the code a bit, to avoid it keeping an ever-growing queue
of failed pushes, if a remote is just dead. And to clear old failed pushes
from the queue when a later push succeeds.

I also need to write a git merge driver that handles conflicts in the tree.
If two conflicting versions of a file `foo` are saved, this would merge
them, renaming them to `foo.X` and `foo.Y`. Probably X and Y are the
git-annex keys for the content of the files; this way all clones will
resolve the conflict in a way that leads to the same tree. It's also
possible to get a conflict by one repo deleting a file, and another
modifying it. In this case, renaming the deleted file to `foo.Y` may
be the right approach, I am not sure.

I glanced through some Haskell dbus bindings today. I belive there are dbus
events available to detect when drives are mounted, and on Linux this would
let git-annex notice and sync to usb drives, etc.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo"
 nickname="Justin"
 subject="comment 1"
 date="2012-06-27T12:46:31Z"
 content="""
can X and Y be the names of the git-annex remotes?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.2.189"
 subject="comment 2"
 date="2012-07-26T17:27:39Z"
 content="""
That's a good question. Unfortunatly they cannot; X and Y need to be stable across repositories, and git remotes can have different names in different repositories.

Even using the description that git-annex stores for each repository for X and Y is problimatic, since that description can change, and so could be different in two repos that are each trying to resolve the same merge conflict.
"""]]


code:
Fixed a goodly amount of bugs today.

The most interesting change was that in direct mode, files using the same
key are no longer hardlinked, as that could cause a surprising behavior if
modifying one, where the other would also change.

Made a release, which is almost entirely bug fixes. Debian amd64 build
included this time. :)