code:
Focus today was writing a notification broadcaster library. This is a way to
send a notification to a set of clients, any of which can be blocked
waiting for a new notification to arrive. A complication is that any number
of clients may be be dead, and we don't want stale notifications for those
clients to pile up and leak memory.

It took me 3 tries to find the solution, which turns out to be head-smackingly
simple: An array of SampleVars, one per client.

Using SampleVars means that clients only see the most recent notification,
but when the notification is just "the assistant's state changed somehow;
display a refreshed rendering of it", that's sufficient.

----

First use of that was to make the thread that woke up every 10 minutes
and checkpointed the daemon status to disk also wait for a notification
that it changed. So that'll be more current, and use less IO.

----

Second use, of course, was to make the WebApp block long polling clients
until there is really a change since the last time the client polled.

To do that, I made one change to my Yesod routes:

[[!format diff """
	-/status StatusR GET
	+/status/#NotificationId StatusR GET
"""]]

Now I find another reason to love Yesod, because after doing that,
I hit "make".. and fixed the type error. And hit make.. and fixed
the type error. And then it just freaking worked! Html was generated with
all urls to /status including a `NotificationId`, and the handler for
that route got it and was able to use it:

[[!format haskell """
	{- Block until there is an updated status to display. -}
	b <- liftIO $ getNotificationBroadcaster webapp
	liftIO $ waitNotification $ notificationHandleFromId b nid
"""]]

And now the WebApp is able to display transfers in realtime!
When I have both the WebApp and `git annex get` running on the same screen,
the WebApp displays files that git-annex is transferring about as fast
as the terminal updates.

The [[progressbars]] still need to be sorted out, but otherwise
the WebApp is a nice live view of file transfers.

---

I also had some fun with Software Transactional Memory. Now when the
assistant moves a transfer from its queue of transfers to do, to its map of
transfers that are currently running, it does so in an atomic transaction.
This will avoid the transfer seeming to go missing (or be listed twice) if
the webapp refreshes at just the wrong point in time. I'm really starting
to get into STM.

----

Next up, I will be making the WebApp maintain a list of notices, displayed
on its sidebar, scrolling new notices into view, and removing ones the user
closes, and ones that expire. This will be used for displaying errors, as
well as other communication with the user (such as displaying a notice
while a git sync is in progress with a remote, etc). Seems worth doing now,
so the basic UI of the WebApp is complete with no placeholders.


code:
Some days I spend 2 hours chasing red herrings (like "perhaps my JSON ajax
calls arn't running asynchronoously?") that turn out to be a simple
one-word typo. This was one of them.

However, I did get the sidebar displaying alert messages, which can be
easily sent to the user from any part of the assistant. This includes
transient alerts of things it's doing, which disappear once the action
finishes, and long-term alerts that are displayed until the user closes
them. It even supports rendering arbitrary Yesod widgets as alerts, so
they can also be used for asking questions, etc.

Time for a screencast!

<video controls src="http://joeyh.name/screencasts/git-annex-webapp.ogg"></video>

repo_name: kakafarm/git

license: unknown


code:
Lots of WebApp UI improvements, mostly around the behavior when
displaying alert messages. Trying to make the alerts informative
without being intrusively annoying, think I've mostly succeeded now.

Also, added an intro display. Shown is the display with only one repo;
if there are more repos it also lists them all.

[[!img screenshot/intro.png]]


code:
Started work on the interface displayed when the webapp is started
with no existing git-annex repository. All this needs to do is walk the user
through setting up a repository, as simply as possible.

A tricky part of this is that most of git-annex runs in the Annex monad,
which requires a git-annex repository. Luckily, much of the webapp
does not run in Annex, and it was pretty easy to work around the parts that
do. Dodged a bullet there.

There will, however, be a tricky transition from this first run webapp,
to a normally fully running git-annex assistant and webapp. I think the
first webapp will have to start up all the normal threads once it makes the
repository, and then redirect the user's web browser to the full webapp.

Anyway, the UI I've made is very simple: A single prompt, for the
directory where the repository should go. With, eventually, tab completion,
sanity checking (putting the repository in "/" is not good, and making it
all of "$HOME" is probably unwise).

[[!img screenshot/firstrun.png]]

Ideally most users will accept the default, which will be something
like `/home/username/Desktop/Annex`, and be through this step in seconds.

Suggestions for a good default directory name appreciated.. Putting it on a
folder that will appear on the desktop seems like a good idea, when there's
a Desktop directory. I'm unsure if I should name it something specific like
"GitAnnex", or something generic like "Synced".

Time for the first of probably many polls!

What should the default directory name used by the git-annex assistant be?

[[!poll open=no 19 "Annex" 7 "GitAnnex" 1 "~/git-annex/" 10 "Synced" 0 "AutoSynced" 1 "Shared" 10 "something lowercase!" 1 "CowboyNeal" 1 "Annexbox"]]

(Note: This is a wiki. You can edit this page to add your own
[[ikiwiki/directive/poll]] options!)

[[!tag polls]]


code:
[[!comment format=mdwn
 username="http://nico.kaiser.me/"
 nickname="Nico Kaiser"
 subject="Directory location"
 date="2012-08-01T07:51:36Z"
 content="""
I don't think the Annex directory (or whatever it will be called – \"Annex\" seems like a good choice in my opinion) should be inside the \"Desktop\" folder by default. All other xdg-user-dirs (Documents, Downloads, etc.) are in $HOME, and i think Desktop should not be cluttered with folders.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.3.236"
 subject="comment 2"
 date="2012-08-01T14:04:08Z"
 content="""
Hmm, my thought on putting it under Desktop is that'll let me install git-annex and tell my mom (really!) that files dragged to there are shared. Don't the other xdg directories get used as default save/open locations for things like word processors, so not need to be on the desktop?

Although that may not be necessary. I've realized I can, at least on linux, easily make a button on the webapp that pops open the directory in the desktop's file manager.

I guess another way of looking at the question is: Where does dropbox put its folder?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawllHWUwOT-UVe5bGkk8G87bpeRAzy-5T7w"
 nickname="Rick"
 subject="Folder location"
 date="2012-08-01T14:18:21Z"
 content="""
I'm of two minds about where the Annex folder should be.

For beginners, the desktop is ideal. No doubt about it. And more experienced users can override the default path during the install.

On the other hand, many of the other 'synced folder' services place their folder in the user's home. Both Dropbox and Skydrive do this. It's where I'd put it for consistency, and to keep my desktop free of clutter.

It's a close call, but if push came to shove, I'd probably go with the user home folder.
"""]]


code:
[[!comment format=mdwn
 username="dhess"
 ip="173.247.200.7"
 subject="Choose a friendly/unintimidating name"
 date="2012-09-13T00:32:15Z"
 content="""
You've already decided to accommodate mom by choosing ~/Desktop as the default location, so you should be consistent: choose a name that means something to people. \"Annex\" and \"GitAnnex\" are good branding, but not very user-friendly. (Contrast with \"Dropbox\", the default folder name for Dropbox -- good branding *and* reasonably meaningful.)

\"Shared\" is friendly, but implies that you're sharing with other people, which isn't necessarily the case. (You should reserve the name \"Shared\" for a sub-directory of the default directory, anyway, if/when the time comes to implement sharing URLs to your g-a-a objects with other users.)

I think that \"Synced\" is fine. Most English-speakers will know what it means, and it's a good description of what g-a-a does.

Don't worry whether we like it or not. Nobody who comments here, nor, likely, anyone who votes, will use the default name anyway ;)
"""]]


code:
Only had a few hours to work today, but my current focus is speed, and I
have indeed sped up parts of `git annex watch`.

One thing folks don't realize about git is that despite a rep for being
fast, it can be rather slow in one area: Writing the index. You don't
notice it until you have a lot of files, and the index gets big. So I've
put a lot of effort into git-annex in the past to avoid writing the index
repeatedly, and queue up big index changes that can happen all at once. The
new `git annex watch` was not able to use that queue. Today I reworked the
queue machinery to support the types of direct index writes it needs, and
now repeated index writes are eliminated.

... Eliminated too far, it turns out, since it doesn't yet *ever* flush
that queue until shutdown! So the next step here will be to have a worker
thread that wakes up periodically, flushes the queue, and autocommits.
(This will, in fact, be the start of the [[syncing]] phase of my roadmap!)
There's lots of room here for smart behavior. Like, if a lot of changes are
being made close together, wait for them to die down before committing. Or,
if it's been idle and a single file appears, commit it immediately, since
this is probably something the user wants synced out right away. I'll start
with something stupid and then add the smarts.

(BTW, in all my years of programming, I have avoided threads like the nasty
bug-prone plague they are. Here I already have three threads, and am going to
add probably 4 or 5 more before I'm done with the git annex assistant. So
far, it's working well -- I give credit to Haskell for making it easy to
manage state in ways that make it possible to reason about how the threads
will interact.)

What about the races I've been stressing over? Well, I have an ulterior
motive in speeding up `git annex watch`, and that's to also be able to
**slow it down**. Running in slow-mo makes it easy to try things that might
cause a race and watch how it reacts. I'll be using this technique when
I circle back around to dealing with the races.

Another tricky speed problem came up today that I also need to fix. On
startup, `git annex watch` scans the whole tree to find files that have
been added or moved etc while it was not running, and take care of them.
Currently, this scan involves re-staging every symlink in the tree. That's
slow! I need to find a way to avoid re-staging symlinks; I may use `git
cat-file` to check if the currently staged symlink is correct, or I may
come up with some better and faster solution. Sleeping on this problem.

----

Oh yeah, I also found one more race bug today. It only happens at startup
and could only make it miss staging file deletions.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawldKnauegZulM7X6JoHJs7Gd5PnDjcgx-E"
 nickname="Matt"
 subject="open source?"
 date="2012-06-09T22:34:30Z"
 content="""
Are you publishing the source code for git-annex assistant somewhere?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.8.126"
 subject="comment 2"
 date="2012-06-09T23:01:29Z"
 content="""
Yes, it's in [[git|download]] with the rest of git-annex. Currently in the `watch` branch.
"""]]


code:
Based on the results of yesterday's poll, the WebApp defaults to
`~/Desktop/annex` when run in the home directory. If there's no `Desktop`
directory, it uses just `~/annex`. And if run from some other place than
the home directory, it assumes you want to use cwd. Of course, you can
change this default, but I think it's a good one for most use cases.

----

My work today has all been on making **one second** of the total lifetime
of the WebApp work. It's the very tricky second in between clicking on
"Make repository" and being redirected to a WebApp running in your new
repository. The trickiness involves threads, and MVars, and
multiple web servers, and I don't want to go into details here.
I'd rather forget. ;-)

Anyway, it works; you can run "git annex webapp" and be walked right
through to having a usable repository! Now I need to see about adding
that to the desktop menus, and making "git annex webapp", when run a second
time, remembering where your repository is. I'll use
`~/.config/git-annex/repository` for storing that.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 1"
 date="2012-08-01T21:19:03Z"
 content="""
Please use `$XDG_CONFIG_HOME` and fall back to `~/.config` if that's not defined.
"""]]


code:
[[!comment format=mdwn
 username="http://piotr.ozarowski.pl/"
 nickname="POX"
 subject="XDG"
 date="2012-08-02T10:30:55Z"
 content="""
same for XDG_DESKTOP_DIR (for pl_PL ~/.config/user-dirs.dirs file contains XDG_DESKTOP_DIR=\"$HOME/Pulpit\" instead of Desktop)
"""]]


code:
Now installing git-annex automatically generates a freedesktop.org .desktop
file, and installs it, either system-wide (root) or locally (user). So
`Menu -> Internet -> Git Annex` will start up the web app.

(I don't entirely like putting it on the Internet menu, but the
Accessories menu is not any better (and much more crowded here),
and there's really no menu where it entirely fits.)

I generated that file by writing a generic library to deal with
freedesktop.org desktop files and locations. Which seemed like overkill at
the time, but then I found myself continuing to use that library. Funny how
that happens.

So, there's also another .desktop file that's used to autostart the
`git-annex assistant` daemon when the user logs into the desktop.

This even works when git-annex is installed to the ugly non-PATH location
`.cabal/bin/git-annex` by Cabal! To make that work, it records the path
the binary is at to a freedesktop.org data file, at install time.

---

That should all work in Gnome, KDE, XFCE, etc. Not Mac OSX I'm guessing...

---

Also today, I added a sidebar notification when the assistant notices new
files. To make that work well, I implemented merging of related sidebar
action notifications, so the effect is that there's one notification that
collectes a list of recently added files, and transient notifications that
show up if a really big file is taking a while to checksum.

I'm pleased that the notification interface is at a point where I was able
to  implement all that, entirely in pure functional code.


code:
Today I added a "Files" link in the navbar of the WebApp. It looks like a
regular hyperlink, but clicking on it opens up your desktop's native file
manager, to manage the files in the repository!

Quite fun to be able to do this kind of thing from a web page. :)

---

Made `git annex init` (and the WebApp) automatically generate a description
of the repo when none is provided.

---

Also worked on the configuration pages some. I don't want to get ahead
of myself by diving into the full configuration stage yet, but I am at
least going to add a configuration screen to clone the repo to a removable
drive.

After that, the list of transfers on the dashboard needs some love.
I'll probably start by adding UI to cancel running transfers, and then
try to get drag and drop reordering of transfers working.


code:
[[!comment format=mdwn
 username="http://cweiske.de/"
 nickname="cweiske"
 subject="comment 1"
 date="2012-08-15T22:02:14Z"
 content="""
I thought that most (if not all) browsers prevent opening local files/directories via web pages. How did you solve that problem?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 subject="comment 2"
 date="2012-08-16T23:34:12Z"
 content="""
Yes, it's surprising to me when I see it work, and I wrote it. :) Since git-annex is running locally on the same system as the browser, it can open the file manager when the user clicks on a link in the web browser. To the web browser, this is just an AJAX request.
"""]]


code:
Spent yesterday and today making the WebApp handle adding removable drives.

While it needs more testing, I think that it's now possible to use the WebApp
for a complete sneakernet usage scenario.

* Start up the webapp, let it make a local repo.
* Add some files, by clicking to open the file manager, and dragging them in.
* Plug in a drive, and tell the webapp to add it.
* Wait while files sync..
* Take the drive to another computer, and repeat the process there.

No command-line needed, and files will automatically be synced between
two or more computers using the drive.

Sneakernet is only one usage scenario for the git-annex assistant, but I'm
really happy to have one scenario 100% working!

Indeed, since the assistant and webapp can now actually do something
useful, I'll probably be merging them into `master` soon.

Details follow..

---

So, yesterday's part of this was building the configuration page to add
a removable drive. That needs to be as simple as possible, and it currently
consists of a list of things git-annex thinks might be mount points of
removable drives, along with how much free space they have. Pick a drive,
click the pretty button, and away it goes..

(I decided to make the page so simple it doesn't even ask where you want
to put the directory on the removable drive. It always puts it in
a "annex" directory. I might add an expert screen later, but experts can
always set this up themselves at the command line too.)

I also fought with Yesod and Bootstrap rather a lot to make the form look good.
Didn't entirely succeed, and had to file a bug on Yesod about its handling of
check boxes. (Bootstrap also has a bug, IMHO; its drop down lists are not
always sized wide enough for their contents.)

Ideally this configuration page would listen for mount events, and refresh
its list. I may add that eventually; I didn't have a handy channel it
could use to do that, so defferred it. Another idea is to have the mount
event listener detect removable drives that don't have an annex on them yet,
and pop up an alert with a link to this configuration page.

----

Making the form led to a somewhat interesting problem: How to tell if a mounted
filesystem is a removable drive, or some random thing like `/proc` or
a fuse filesystem. My answer, besides checking that the user can
write to it, was various heuristics, which seem to work ok, at least here..

[[!format haskell """
               sane Mntent { mnt_dir = dir, mnt_fsname = dev }
                        {- We want real disks like /dev/foo, not
                         - dummy mount points like proc or tmpfs or
                         - gvfs-fuse-daemon. -}
                        | not ('/' `elem` dev) = False
                        {- Just in case: These mount points are surely not
                         - removable disks. -}
                        | dir == "/" = False
                        | dir == "/tmp" = False
                        | dir == "/run/shm" = False
                        | dir == "/run/lock" = False
"""]]

----

Today I did all the gritty coding to make it create a git repository on the
removable drive, and tell the Annex monad about it, and ensure it gets synced.

As part of that, it detects when the removable drive's filesystem doesn't
support symlinks, and makes a bare repository in that case. Another expert
level config option that's left out for now is to always make a bare
repository, or even to make a directory special remote rather than a git
repository at all. (But directory special remotes cannot support the
sneakernet use case by themselves...)

----

Another somewhat interesting problem was what to call the git remotes
that it sets up on the removable drive and the local repository.
Again this could have an expert-level configuration, but the defaults
I chose are to use the hostname as the remote name on the removable drive,
and to use the basename of the mount point of the removable drive as the
remote name in the local annex.

----

Originally, I had thought of this as cloning the repository to the drive.
But, partly due to luck, I started out just doing a `git init` to make
the repository (I had a function lying around to do that..).

And as I worked on it some more, I realized this is not as simple as a
clone. It's a bi-directional sync/merge, and indeed the removable drive may
have all the data already in it, and the local repository have just been
created. Handling all the edge cases of that (like, the local repository
may not have a "master" branch yet..) was fun!


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 1"
 date="2012-08-05T23:50:57Z"
 content="""
Using `annex` as default is fine, but even non-technical users will need a way to create different annexes for different usage. Or if two users share one thumb drive and each has their own annex. Long story short, I don't think this is an expert option, but something every user should be able to change immediately.

A pre-populated text area makes most sense, imo.
"""]]


code:
[[!comment format=mdwn
 username="http://dieter-be.myopenid.com/"
 nickname="dieter"
 subject="comment 2"
 date="2012-08-06T08:30:47Z"
 content="""
+1 on what Richard said
about the correctly figuring out what are valid mountpoints / devices that can be used, your current code doesn't seem very robust.
check this out https://github.com/Dieterbe/aif/blob/develop/src/core/libs/lib-blockdevices-filesystems.sh#L213
"""]]


code:
Nothing flashy today; I was up all night trying to download photos taken
by a robot lowered onto Mars by a skycrane.

Some work on alerts. Added an alert when a file transfer succeeds or fails.
Improved the alert combining code so it handles those alerts, and
simplified it a lot, and made it more efficient.

Also made the text of action alerts change from present to past tense when
the action finishes. To support that I wrote a fun data type, a `TenseString`
that can be rendered in either tense.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlJ2utMQgMEYAOs3Dfc6eZRyUzt4acNXUU"
 nickname="David"
 subject="comment 1"
 date="2012-08-06T23:48:28Z"
 content="""
I see you are a regular XKCD reader
"""]]


code:
A bit under the weather, but got into building buttons to control running
and queued transfers today. The html and javascript side is done, with
each transfer now having a cancel button, as well as a pause/start button.

Canceling queued transfers works. Canceling running transfers will
need some more work, because killing a thread doesn't kill the processes
being run by that thread. So I'll have to make the assistant run separate
git-annex processes for transfers, that can be individually sent signals.


code:
Probably won't be doing any big coding on the git-annex assistant in the
upcoming week, as I'll be traveling and/or slightly ill enough that I can't
fully get into flow.

---

There was a new Yesod release this week, which required minor changes to
make the webapp build with it. I managed to keep the old version of Yesod
also supported, and plan to keep that working so it can be built with the
version of Yesod available in, eg, Linux distributions. TBD how much pain
that will involve going forward.

---

I'm mulling over how to support stopping/pausing transfers. The problem
is that if the assistant is running a transfer in one thread, and the
webapp is used to cancel it, killing that thread won't necessarily stop the
transfer, because, at least in Haskell's thread model, killing a thread
does not kill processes started by the thread (like rsync).

So one option is to have the transfer thread run a separate git-annex
process, which will run the actual transfer. And killing that process will
stop the transfer nicely. However, using a separate git-annex process means
a little startup overhead for each file transferred (I don't know if it'd
be enough to matter). Also, there's the problem that git-annex is sometimes
not installed in PATH (wish I understood why cabal does that), which
makes it kind of hard for it to run itself. (It can't simply fork, sadly.
See past horrible pain with forking and threads.)

The other option is to change the API for git-annex remotes, so that
their `storeKey` and `retrieveKeyFile` methods return a pid of the program
that they run. When they *do* run a program.. not all remotes do. This
seems like it'd make the code in the remotes hairier, and it is also asking
for bugs, when a remote's implementation changes. Or I could go
lower-level, and make every place in the utility libraries that forks a
process record its pid in a per-thread MVar. Still seems to be asking for
bugs.

Oh well, at least git-annex is already crash-safe, so once I figure out
how to kill a transfer process, I can kill it safely. :)


code:
[[!comment format=mdwn
 username="http://claimid.com/strager"
 nickname="strager"
 subject="comment 1"
 date="2012-08-11T04:50:52Z"
 content="""
What if `storeKey`, `retrieveKeyFile`, etc. return an `IO ()` which cancels the operation, if possible?  The implementation can be canceled regardless if it uses separate processes or Haskell threads.

"""]]


code:
[[!comment format=mdwn
 username="http://claimid.com/strager"
 nickname="strager"
 subject="comment 2"
 date="2012-08-11T04:55:13Z"
 content="""
In fact, making a dedicated data type or some typeclasses may be more appropriate:

    class Cancelable a where cancel :: a -> IO ()
    class Pauseable a where pause :: a -> IO ()

    -- Alternatively:

    data Transfer = Transfer { cancel :: IO (), pause :: IO () }

    -- Or both!

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 subject="comment 3"
 date="2012-08-11T14:41:51Z"
 content="""
That's the lines I was thinking along, and I even made a throwaway branch with some types. But the problem is reworking all the code to do that. Particularly since lots of the code uses generic utility functions that are reused in other, unrelated places and would have to be modified to pass back cancel actions.

The first case the type checker landed me on when I changed the types was code that downloads an url from the web. Naturally that uses a Utility.Url.download. How to cancel `download`? Depends on its implementation -- it happens to currently shell out to curl, so you have to kill curl, but it could just as easily have used libcurl (other parts of my Utility.Url library do), and then it would need to fork its own thread. So it's an abstraction layer violation problem.

If I had a month to devote to this one problem, I might manage to come up with some clean solution involving monads, or maybe convert all my code to use conduit or something that might allow managing these effects better. Just a guess..
"""]]


code:
[[!comment format=mdwn
 username="http://claimid.com/strager"
 ip="173.228.13.253"
 subject="comment 4"
 date="2012-08-11T16:08:47Z"
 content="""
> How to cancel download?  Depends on its implementation ....  So it's an abstraction layer violation problem.

Precisely why I suggested returning something as generic as `IO ()`:

    -- Current
    download :: URLString -> Headers -> [CommandParam] -> FilePath -> IO Bool

    -- Suggestion
    data Transfer a = Transfer { run :: IO a, cancel :: IO () }
    download :: URLString -> Headers -> [CommandParam] -> FilePath -> Transfer

    transfer <- download ...
    -- You can pass `cancel transfer` to another thread
    -- which you want to be able to cancel the transfer.
    run transfer  -- blocking

I realized while writing this that you may not get any result from e.g. a download while it is occurring (because the function is blocking).  Maybe that's where a misunderstanding occurred.  I separated the concepts of creating a transfer and starting/canceling it.

(My idea is starting to feel a bit object-oriented...  ;P)

"""]]


code:
Unexpectedly managed a mostly productive day today.

Went ahead with making the assistant run separate `git-annex` processes for
transfers. This will currently fail if git-annex is not installed in PATH.
(Deferred dealing with that.)

To stop a transfer, the webapp needs to signal not just the git-annex
process, but all its children. I'm using process groups for this, which is
working, but I'm not extremely happy with.

Anyway, the webapp's UI can now be used for stopping transfers, and it
wasn't far from there to also implementing pausing of transfers.

Pausing a transfer is actually the same as stopping it, except a special
signal is sent to the transfer control thread, which keeps running, despite
the git-annex process having been killed, waits for a special resume
signal, and restarts the transfer. This way a paused transfer continues to
occupy a transfer slot, which prevents other queued transfers from running.
This seems to be the behavior that makes sense.

Still need to wire up the webapp's button for starting a transfer. For a
paused transfer, that will just need to resume it. I have not decided what
the button should do when used on a transfer that is queued but not running
yet. Maybe it forces it to run even if all transfer slots are already in
use? Maybe it stops one of the currently running transfers to free up a
slot?


code:
Actually did do some work on the webapp today, just fixing a bug I noticed
in a spare moment. Also managed a bit in the plane earlier this week,
implementing resuming of paused transfers. (Still need to test that.)

But the big thing today was dinner with one of my major Kickstarter
backers, and as it turned out, "half the Haskell community of San
Francisco" (3 people). Enjoyed talking about git-annex and haskell with
them.

I'm looking forward to getting back home and back to work on Monday..


code:
[[!comment format=mdwn
 username="http://claimid.com/strager"
 ip="173.228.13.253"
 subject="comment 1"
 date="2012-08-17T15:59:49Z"
 content="""
It was fun to have you here!  =]

"""]]


code:
After a few days otherwise engaged, back to work today.

My focus was on adding the committing thread mentioned in [[day_4__speed]].
I got rather further than expected!

First, I implemented a really dumb thread, that woke up once per second,
checked if any changes had been made, and committed them. Of course, this
rather sucked. In the middle of a large operation like untarring a tarball,
or `rm -r` of a large directory tree, it made lots of commits and made
things slow and ugly. This was not unexpected.

So next, I added some smarts to it. First, I wanted to stop it waking up
every second when there was nothing to do, and instead blocking wait on a
change occurring. Secondly, I wanted it to know when past changes happened,
so it could detect batch mode scenarios, and avoid committing too
frequently.

I played around with combinations of various Haskell thread communications
tools to get that information to the committer thread: `MVar`, `Chan`,
`QSem`, `QSemN`. Eventually, I realized all I needed was a simple channel
through which the timestamps of changes could be sent. However, `Chan`
wasn't quite suitable, and I had to add a dependency on
[Software Transactional Memory](http://en.wikipedia.org/wiki/Software_Transactional_Memory),
and use a `TChan`. Now I'm cooking with gas!

With that data channel available to the committer thread, it quickly got
some very nice smart behavior. Playing around with it, I find it commits
*instantly* when I'm making some random change that I'd want the
git-annex assistant to sync out instantly; and that its batch job detection
works pretty well too.

There's surely room for improvement, and I made this part of the code
be an entirely pure function, so it's really easy to change the strategy.
This part of the committer thread is so nice and clean, that here's the
current code, for your viewing pleasure:

[[!format haskell """
{- Decide if now is a good time to make a commit.
 - Note that the list of change times has an undefined order.
 -
 - Current strategy: If there have been 10 commits within the past second,
 - a batch activity is taking place, so wait for later.
 -}
shouldCommit :: UTCTime -> [UTCTime] -> Bool
shouldCommit now changetimes
       | len == 0 = False
       | len > 4096 = True -- avoid bloating queue too much
       | length (filter thisSecond changetimes) < 10 = True
       | otherwise = False -- batch activity
       where
               len = length changetimes
               thisSecond t = now `diffUTCTime` t <= 1
"""]]

Still some polishing to do to eliminate minor inefficiencies and deal
with more races, but this part of the git-annex assistant is now very usable,
and will be going out to my beta testers soon!


code:
As I prepare to dive back into development, now is a good time to review
what I've built so far, and how well I'm keeping up with my planned
[[roadmap|assistant]].

I started working two and a half months ago, so am nearing the end of
the three months I originally asked to be funded for on Kickstarter.

I've built much of what I planned to build in the first three months --
[[inotify]] is done (and kqueue is basically working, but needs scalability
work), local [[syncing]] is done, the [[webapp]] works, and I've built some
of the first [[configurators]]. It's all functional in a narrow use case
involving syncing to removable drives.

[[progressbars]] still need to be dealt with, and network syncing needs to
be revisited soon, so that I can start building easy configurators for
further use cases, like using the cloud, or another machine on the local
network.

I think I'm a little behind my original schedule, but not too bad,
and at the same time, I think I've built things rather more solidly than I
expected them to be at this point. I'm particularly happy with how well
the inotify code works, no matter what is thrown at it, and how nice
the UI in the webapp is shaping up to be.

----

I also need to get started on fulfilling my Kickstarter rewards, and
I was happy to spend some time in the airport working on the main
blocker toward that, a lack of a scalable git-annex logo, which is needed
for printing on swag.

Turns out that inkscape has some amazing bitmap tracing capabilities.
I was able to come up with this scalable logo in short order, it
actually took longer to add back the colors, as the tracer generated a
black and white version.

With that roadblock out of the way, I am moving toward ordering large
quantities of usb drives, etc.

[[logo.svg]]


code:
[[!comment format=mdwn
 username="http://dieter-be.myopenid.com/"
 nickname="dieter"
 subject="better logo?"
 date="2012-08-20T21:22:25Z"
 content="""
Maybe you should contact designers and see what kind of logo they can come up with.  Probably something better.
"""]]


code:
Today, added a thread that deals with recovering when there's been a loss
of network connectivity. When the network's down, the normal immediate
syncing of changes of course doesn't work. So this thread detects when the
network comes back up, and does a pull+push to network remotes, and
triggers scanning for file content that needs to be transferred.

I used dbus again, to detect events generated by both network-manager and
wicd when they've sucessfully brought an interface up. Or, if they're not
available, it polls every 30 minutes.

When the network comes up, in addition to the git pull+push, it also
currently does a full scan of the repo to find files whose contents
need to be transferred to get fully back into sync.

I think it'll be ok for some git pulls and pushes to happen when
moving to a new network, or resuming a laptop (or every 30 minutes when
resorting to polling). But the transfer scan is currently really too heavy
to be appropriate to do every time in those situations. I have an idea for
avoiding that scan when the remote's git-annex branch has not changed. But
I need to refine it, to handle cases like this:

1. a new remote is added
2. file contents start being transferred to (or from it)
3. the network is taken down
4. all the queued transfers fail
5. the network comes back up
6. the transfer scan needs to know the remote was not all in sync
   before #3, and so should do a full scan despite the git-annex branch
   not having changed

---

Doubled the ram in my netbook, which I use for all development. Yesod needs
rather a lot of ram to compile and link, and this should make me quite a
lot more productive. I was struggling with OOM killing bits of chromium
during my last week of development.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmubB1Sj2rwFoVdZYvGV0ACaQUJQyiJXJI"
 nickname="Paul"
 subject="Amazon Glacier"
 date="2012-08-23T06:32:24Z"
 content="""
Do you think git-annex could support [Amazon Glacier](http://aws.amazon.com/glacier/) as a backend?
"""]]


code:
Woke up this morning with most of the design for a smarter approach to
[[syncing]] in my head. (This is why I sometimes slip up and tell people I
work on this project 12 hours a day..)

To keep the current `assistant` branch working while I make changes
that break use cases that are working, I've started
developing in a new branch, `assistant-wip`.

In it, I've started getting rid of unnecessary expensive transfer scans.

First optimisation I've done is to detect when a remote that was
disconnected has diverged its `git-annex` branch from the local branch.
Only when that's the case does a new transfer scan need to be done, to find
out what new stuff might be available on that remote, to have caused the
change to its branch, while it was disconnected.

That broke a lot of stuff. I have a plan to fix it written down in
[[syncing]]. It'll involve keeping track of whether a transfer scan has
ever been done (if not, one should be run), and recording logs when
transfers failed, so those failed transfers can be retried when the
remote gets reconnected.


code:
Implemented everything I planned out yesterday: Expensive scans are only
done once per remote (unless the remote changed while it was disconnected),
and failed transfers are logged so they can be retried later.

Changed the TransferScanner to prefer to scan low cost remotes first,
as a crude form of scheduling lower-cost transfers first.

A whole bunch of interesting syncing scenarios should work now. I have not
tested them all in detail, but to the best of my knowledge, all these
should work:

* Connect to the network. It starts syncing with a networked remote.
  Disconnect the network. Reconnect, and it resumes where it left off.
* Migrate between networks (ie, home to cafe to work). Any transfers
  that can only happen on one LAN are retried on each new network you
  visit, until they succeed.

One that is not working, but is soooo close:

* Plug in a removable drive. Some transfers start. Yank the plug.
  Plug it back in. All necessary transfers resume, and it ends up
  fully in sync, no matter how many times you yank that cable.

That's not working because of an infelicity in the MountWatcher.
It doesn't notice when the drive gets unmounted, so it ignores
the new mount event.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo"
 nickname="Justin"
 subject="comment 1"
 date="2012-08-23T21:25:48Z"
 content="""
Do encrypted rsync remotes resume quickly as well?

One thing I noticed was that if a copy --to an encrypted rsync remote gets interrupted it will remove the tmp file and re-encrypt the whole file before resuming rsync.
"""]]


code:
Working toward getting the data syncing to happen robustly,
so a bunch of improvements.

* Got unmount events to be noticed, so unplugging and replugging
  a removable drive will resume the syncing to it. There's really no
  good unmount event available on dbus in kde, so it uses a heuristic
  there.
* Avoid requeuing a download from a remote that no longer has a key.
* Run a full scan on startup, for multiple reasons, including dealing with
  crashes.

Ran into a strange issue: Occasionally the assistant will run `git-annex
copy` and it will not transfer the requested file. It seems that
when the copy command runs `git ls-files`, it does not see the file
it's supposed to act on in its output.

Eventually I figured out what's going on: When updating the git-annex
branch, it sets `GIT_INDEX_FILE`, and of course environment settings are
not thread-safe! So there's a race between threads that access
the git-annex branch, and the Transferrer thread, or any other thread
that might expect to look at the normal git index.

Unfortunatly, I don't have a fix for this yet.. Git's only interface for
using a different index file is `GIT_INDEX_FILE`. It seems I have a lot of
code to tear apart, to push back the setenv until after forking every git
command. :(

Before I figured out the root problem, I developed a workaround for the
symptom I was seeing. I added a `git-annex transferkey`, which is
optimised to be run by the assistant, and avoids running `git ls-files`, so
avoids the problem. While I plan to fix this environment variable problem
properly, `transferkey` turns out to be so much faster than how it was
using `copy` that I'm going to keep it.


code:
Almost done with the data transfer code.. Today I filled in some bits and
peices.

Made the expensive transfer scan handle multiple remotes in one pass.
So on startup, it only runs once, not N times. And when reconnecting to the
network, when a remote has changed, it scans all network remotes in one
pass, rather than making M redundant passes.

Got syncing with special remotes all working. Pretty easy actually. Just
had to avoid doing any git repo push/pull with them, while still queueing
data transfers.

It'll even download anything it can from the web special remote. To support
that, I added generic support for readonly remotes; it'll only download from
those and not try to upload to them.

(Oh, and I properly fixed the nasty `GIT_INDEX_FILE` environment variable
problem I had the other day.)

I feel I'm very close to being able to merge the assistant branch into
master now. I'm reasonably confident the data transfer code will work
well now, and manage to get things in sync eventually in all circumstances.
(Unless there are bugs.) All the other core functionality of the assistant
and webapp is working. The only think I might delay because of is the
missing [[progressbars]] in the webapp .. but that's a silly thing to
block the merge on.

Still, I might spend a day and get a dumb implementation of progress bars
for downloads working first (progress bars for uploads are probably rather
harder). I'd spend longer on progress bars, but there are so many more
exciting things I'm now ready to develop, like automatic configurators
for using your git annex with Amazon S3, rsync.net, and the computer across
the room..!


code:
It's done! The assistant branch is merged into master.

Updated the [[assistant]] page with some screenshots and instructions for
using it.

Made some cosmetic fixes to the webapp.

Fixed the transferrer to use `~/.config/git-annex/program`
to find the path to git-annex when running it. (There are ways to find the
path of the currently running program in unux, but they all suck, so I'm
avoiding them this way.)

Read some OSX launchd documentation, and it seems it'd be pretty easy to
get the assistant to autostart on login on OSX. If someone would like to
test launchd files for me, get in touch.

-----

AKA: Procrastinating really hard on those progress bars. ;)


code:
[[!comment format=mdwn
 username="http://kevwalke.myopenid.com/"
 ip="94.175.38.180"
 subject="confirmed!"
 date="2012-09-02T07:17:43Z"
 content="""
Got it built and running. Thanks for this.
I can see already that this is going to be a fantastic application.

Good Work!
-Kev
"""]]


code:
[[!comment format=mdwn
 username="http://kevwalke.myopenid.com/"
 ip="94.175.38.180"
 subject="well done!"
 date="2012-08-27T22:08:23Z"
 content="""
absolutely fantastic news! I am itching to try this out now. Thank you!
"""]]


code:
[[!comment format=mdwn
 username="http://claimid.com/strager"
 ip="173.228.13.253"
 subject="comment 2"
 date="2012-08-28T16:29:56Z"
 content="""
I have a few OS X machines I can test on.  Send me details at: strager.nds@gmail.com
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 3"
 date="2012-08-28T18:44:28Z"
 content="""
As per usual I'd volunteer to test on OSX as well.
"""]]


code:
[[!comment format=mdwn
 username="http://kevwalke.myopenid.com/"
 ip="77.86.30.139"
 subject="building and installing with cabal"
 date="2012-08-29T18:15:07Z"
 content="""
So I got round to trying it out. I cloned the latest from the repository@ git://git-annex.branchable.com/ and then followed the instructions on the install page: http://git-annex.branchable.com/install/cabal/
However I get a dependency problem:

cabal install --only-dependencies
Resolving dependencies...
cabal: Could not resolve dependencies:
trying: git-annex-3.20120825 (user goal)
trying: git-annex-3.20120825:+oldyesod
trying: git-annex-3.20120825:+currentyesod
next goal: yesod-default (dependency of git-annex-3.20120825:+oldyesod)
rejecting: yesod-default-1.1.0 (conflict: git-annex-3.20120825:oldyesod =>
yesod-default(<=1.0.1.1))
rejecting: yesod-default-1.0.1.1, 1.0.1, 1.0.0, 0.6.1, 0.5.0, 0.4.1, 0.4.0,
0.3.1 (conflict: git-annex-3.20120825:currentyesod => yesod-default(>=1.1.0))

I'm not sure how to proceed from here and would be grateful of any pointers to help get this built
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.246.16"
 subject="comment 5"
 date="2012-08-29T22:17:01Z"
 content="""
@kevwalke: I fixed that problem today.
"""]]


code:
[[!comment format=mdwn
 username="http://kevwalke.myopenid.com/"
 ip="77.86.30.139"
 subject="getting new error now."
 date="2012-08-30T11:55:49Z"
 content="""
Seems there is still an issue here on fresh install...

cabal: Error: some packages failed to install:
dbus-0.10 depends on libxml-sax-0.7.2 which failed to install.
libxml-sax-0.7.2 failed during the configure step. The exception was:
ExitFailure
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.246.16"
 subject="comment 7"
 date="2012-08-30T17:15:58Z"
 content="""
It's hard to tell from that error message what's going on. I'd hope that there was actually a real error message printed earlier than that that.

For now, I can only guess.. In order to install the haskell libxml-sax library, you need to have the GNOME XML2 library installed, including development headers. On Debian and derivatives, that is `apt-get install libxml2-dev`.

Alternatively, you can disable use of dbus by passing this to your cabal install: `--flags=-Dbus`
"""]]


code:
[[!comment format=mdwn
 username="http://kevwalke.myopenid.com/"
 ip="94.175.38.180"
 subject="thanks"
 date="2012-09-01T14:44:58Z"
 content="""
That got rid of that error. However now I get the following at the last step:

Linking dist/build/git-annex/git-annex ...
Installing executable(s) in /home/kev/bin
setup: git-annex.1: does not exist
cabal: Error: some packages failed to install:
git-annex-3.20120826 failed during the final install step. The exception was:
ExitFailure 1

There doesn't seem to be any failures further up the output just some warnings. All the other steps complete fine
Thanks for any advice/pointers here
-Kevin
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.246.16"
 subject="comment 9"
 date="2012-09-01T19:32:02Z"
 content="""
I think I've fixed that.
"""]]


code:
Got the webapp's progress bars updating for downloads. Updated
[[progressbars]] with all the options for ways to get progress info. For
downloads, it currently uses the easy, and not very expensive, approach of
periodically polling the sizes of files that are being downloaded.

For uploads, something more sophisticated will be called for..

---

The webapp really feels alive now that it has progress bars!


code:
More work on the display and control of transfers.

* Hide redundant downloads from the transfer display. It seemed simplest
  to keep the behavior of queuing downloads from every remote that has a
  file, rather than going to some other data structure, but it's clutter
  to display those to the user, especially when you often have 7 copies
  of each file, like I do.
* When canceling a download, cancel all other queued downloads of that
  key too.
* Fixed unsettting of the paused flag when resuming a paused transfer.
* Implemented starting queued transfers by clicking on the start button.
* Spent a long time debugging why pausing, then resuming, and then pausing
  a transfer doesn't successfully pause it the second time. I see where
  the code is seemingly locking up in a `throwTo`, but I don't understand
  why that blocks forever. Urgh..


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 1"
 date="2012-08-29T21:24:37Z"
 content="""
It seems to be a bad idea to cancel everything when I cancel a single transfer.

I may not want to transmit videos and RAW files at the moment, but would still like to know my JPGs are stored in a second location.

Two obvious approaches, ideally both offered at the same time would be to allow the user to cancel, cancel all to this remote, and cancel all plus allowing them to send transfers to the end of the queue.

If this is implemented already then sorry; I did _not_ test if this is available before writing this comment.
"""]]


code:
Short day today.

* Worked on fixing a number of build failures people reported.
* Solved the problem that was making transfer pause/resume not always work.
  Although there is another bug where pausing a transfer sometimes lets
  another queued transfer start running.
* Worked on getting the assistant to start on login on OSX.


code:
Since my last blog, I've been polishing the `git annex watch` command.

First, I fixed the double commits problem. There's still some extra
committing going on in the `git-annex` branch that I don't understand. It
seems like a shutdown event is somehow being triggered whenever
a git command is run by the commit thread.

I also made `git annex watch` run as a proper daemon, with locking to
prevent multiple copies running, and a pid file, and everything.
I made `git annex watch --stop` stop it.

---

Then I managed to greatly increase its startup speed. At startup, it
generates "add" events for every symlink in the tree. This is necessary
because it doesn't really know if a symlink is already added, or was
manually added before it starter, or indeed was added while it started up.
Problem was that these events were causing a lot of work staging the
symlinks -- most of which were already correctly staged.

You'd think it could just check if the same symlink was in the index.
But it can't, because the index is in a constant state of flux. The
symlinks might have just been deleted and re-added, or changed, and
the index still have the old value.

Instead, I got creative. :) We can't trust what the index says about the
symlink, but if the index happens to contain a symlink that looks right,
we can trust that the SHA1 of its blob is the right SHA1, and reuse it
when re-staging the symlink. Wham! Massive speedup!

---

Then I started running `git annex watch` on my own real git annex repos,
and noticed some problems.. Like it turns normal files already checked into
git into symlinks. And it leaks memory scanning a big tree. Oops..

---

I put together a quick screencast demoing `git annex watch`.

<video controls src="http://joeyh.name/screencasts/git-annex-watch.ogg"></video>

While making the screencast, I noticed that `git-annex watch` was spinning
in strace, which is bad news for powertop and battery usage. This seems to
be a [GHC bug](http://bugs.debian.org/677096) also affecting Xmonad. I
tried switching to GHC's threaded runtime, which solves that problem, but
causes git-annex to hang under heavy load. Tried to debug that for quite a
while, but didn't get far. Will need to investigate this further..
Am seeing indications that this problem only affects ghc 7.4.1; in
particular 7.4.2 does not seem to have the problem.


code:
Today I built the UI in the webapp to set up a ssh or rsync remote.

This is the most generic type of remote, and so it's surely got the most
complex description. I've tried to word it as clearly as I can; suggestions
most appreciated. Perhaps I should put in a diagram?

[[!img /assistant/addsshserver.png]]

The idea is that this will probe the server, using ssh. If `git-annex-shell`
is available there, it'll go on to set up a full git remote. If not, it'll
fall back to setting up a rsync special remote. It'll even fall all the way
back to using `rsync://` protocol if it can't connect by ssh. So the user
can just point it at a server and let it take care of the details,
generally.

The trickiest part of this will be authentication, of course. I'm relying
on ssh using `ssh-askpass` to prompt for any passwords, etc, when there's
no controlling terminal. But beyond passwords, this has to deal with ssh
keys.

I'm planning to make it check if you have a ssh key configured already. If
you do, it doesn't touch your ssh configuration. I don't want to get in the
way of people who have a manual configuration or are using MonkeySphere.

But for a user who has never set up a ssh key, it will prompt asking if
they'd like a key to be set up. If so, it'll generate a key and configure
ssh to only use it with the server.. and as part of its ssh probe, that key
will be added to `authorized_keys`.

(Obviously, advanced users can skip this entirely; `git remote add
ssh://...` still works..)

----

Also today, fixed more UI glitches in the transfer display. I think
I have them all fixed now, except for the one that needs lots of javascript
to be written to fix it.

Amusingly, while I was working on UI glitches, it turned out that all the
fixes involved 100% pure code that has nothing to do with UI. The UI was
actually just exposing bugs.

For example, closing a running transfer
had a bug that weirdly reordered the queue. This turned out to be
due to the transfer queue actually maintaining two versions of the queue,
one in a TChan and one a list. Some unknown bugs caused these to get out of
sync. That was fixed very handily by deleting the TChan, so there's only
one copy of the data.

I had only been using that TChan because I wanted a way to block while the
queue was empty. But now that I'm more comfortable with STM, I know how
to do that easily using a list:

[[!format haskell """
	getQueuedTransfer q = atomically $ do
		sz <- readTVar (queuesize q)
		if sz < 1
			then retry -- blocks until size changes
			else ...
"""]]

Ah, the times before [STM](http://en.wikipedia.org/wiki/Software_transactional_memory)
were dark times indeed. I'm writing more and more STM code lately, building
up more and more complicated and useful transactions. If you use threads and
don't know about STM, it's a great thing to learn, to get out of the dark ages
of dealing with priority inversions, deadlocks, and races.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmLB39PC89rfGaA8SwrsnB6tbumezj-aC0"
 nickname="Tobias"
 subject="comment 10"
 date="2012-09-09T16:58:55Z"
 content="""
That's great news.

I suggest we disregard this comment thread, i'll just follow the proper bug report(which i guess i should have made to begin with).

Thanks for your good work.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.8.30"
 subject="comment 11"
 date="2012-09-09T17:12:55Z"
 content="""
That bug should be fixed now.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmLB39PC89rfGaA8SwrsnB6tbumezj-aC0"
 nickname="Tobias"
 subject="gpg encryption on ssh remotes"
 date="2012-09-01T08:25:17Z"
 content="""
I've been playing around with the Assistant in the last few days.

And besides for gpg encryption it works great.

But whenever i set up a special remote(done it manually since you hadn't made this yet) it will break, bad.

Some files will transfer fine, but at some point it will still(the stdout from git-annex webapp shows gpg being stupid).

I get a huge amount of [git] <defunct> in my ps auxw. And two git-annex transferkey <key> --to host --file <some file>

It seems to consistently fall on a 9mb file i have. I have found no way to recover from this(killing and restarting git-annex will say 'transfer already in progress'. Deleting the file was to no avail, it still tries to transfer).

I have done a lot of testing and i've only seen this on special remotes when gpg encryption is enabled(i haven't tried encryption=shared)

Besides for this issue, the Assistant is looking great.

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.236"
 subject="comment 2"
 date="2012-09-02T21:40:17Z"
 content="""
@tobias, when you say \"gpg being stupid\", what exactly do you see at that point? Please paste it.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmLB39PC89rfGaA8SwrsnB6tbumezj-aC0"
 nickname="Tobias"
 subject="comment 3"
 date="2012-09-02T23:49:17Z"
 content="""
Updated to the latest trunk, and did some more testing.

I'll retract the \"gpg being stupid\" comment. it just gives the normal \"(gpg)\".

There really isn't anything interesting in the stdout(but a lot of defunct git's in ps auxw), but i still can't get any files larger than ~710kbyte to transfer. They just stall.

http://pastebin.com/ZA8dzxZD <- stdout/stderr
http://pastebin.com/f1J1T79E <- ps auxw | grep git

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmLB39PC89rfGaA8SwrsnB6tbumezj-aC0"
 nickname="Tobias"
 subject="comment 4"
 date="2012-09-03T08:12:39Z"
 content="""
So, i've let the assistant run since i posted my last message, while i slept.

But 8 hours later, it still hasn't made any progress. :(

Again, only see this with gpg encryption enabled, and on files >700kbyte.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.236"
 subject="comment 5"
 date="2012-09-04T20:06:17Z"
 content="""
Can you try running the transfer command by hand? The command line is:

`/home/alansmithee/bin/git-annex transferkey SHA256-s806100--ad2c33a88a665c56d97393fad99b37529afc5947694eab7d9c27f4d178d182f4 --to host1free --file 787kbyte.txt`

Paste me the full output until it seems to hang (if it hangs this way), and if it does hang you may also need to strace it and send me that..
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmLB39PC89rfGaA8SwrsnB6tbumezj-aC0"
 nickname="Tobias"
 subject="comment 6"
 date="2012-09-04T21:57:05Z"
 content="""
No joy.

strace: http://pastebin.com/UYSrZZAg
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.236"
 subject="comment 7"
 date="2012-09-06T16:20:49Z"
 content="""
From the strace, I can see it reading the file and feeding it to gpg. Either gpg is stalling after a while, or perhaps it is not reading the output back from gpg. What type of special remote is this (rsync/directory/S3)? Can you provide `strace -f` output for the same command you straced before?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmLB39PC89rfGaA8SwrsnB6tbumezj-aC0"
 nickname="Tobias"
 subject="comment 8"
 date="2012-09-06T18:41:47Z"
 content="""
I've tried with rsync and directory(real, mountpoint, and fuse mount), no S3. Same result on all of them.

Here is an strace -f http://pastebin.com/YqCbi7Ue

Funny thing is that now i hitting the failure on smaller files(<500kbyte).

I even did a restart because of that. To see if a clean boot would increate it back to the ~700kbyte, but it made no difference.

The annex hasn't been idle since last time, but i haven't done anything major in it. In total the annex is 204mb(up from 185mb at the time of my first post about this)
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.8.30"
 subject="comment 9"
 date="2012-09-09T16:52:33Z"
 content="""
Someone else reported what looks like the same problem, here: [[bugs/transferkey_fails_due_to_gpg]]

I'mm be following up to that bug report. In fact, I think I just managed to replicate the bug!
"""]]


code:
Got ssh probing implemented. It checks if it can connect to the server, and
probes the server to see how it should be used.

Turned out to need two ssh probes. The first uses the system's existing ssh
configuration, but disables password prompts. If that's able to get in
without prompting for a password, then the user must have set that up,
and doesn't want to be bothered with password prompts, and it'll respect
that configuration.

Otherwise, it sets up a per-host ssh key, and configures a hostname alias
in `~/.ssh/config` to use that key, and probes using that.
Configuring ssh this way is nice because it avoids changing ssh's
behavior except when git-annex uses it, and it does not open up the server
to arbitrary commands being run without password.

--

Next up will be creating the repositories. When there's a per-host key,
this will also involve setting up `authorized_keys`, locking down the ssh
key to only allow running git-annex-shell or rsync.

I decided to keep that separate from the ssh probing, even though it means
the user will be prompted twice for their ssh password. It's cleaner and
allows the probing to do other checks -- maybe it'll later check the amount
of free disk space -- and the user should be able to decide after the probe
whether or not to proceed with making the repository.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmRFKwny4rArBaz-36xTcsJYqKIgdDaw5Q"
 nickname="Andrew"
 subject="comment 1"
 date="2013-01-19T07:23:38Z"
 content="""
I'm finding a few circumstances where the per-host key never gets setup. Is there a way to force this probing or key setup to happen?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.3.194"
 subject="comment 2"
 date="2013-01-19T16:09:15Z"
 content="""
It'd be best to file a bug report with details.
"""]]


code:
Decided to only make bare git repos on remote ssh servers. This
configurator is aimed at using a server somewhere, which is probably not
going to be running the assistant. So it doesn't need a non-bare repo, and
there's nothing to keep the checked out branch in a non-bare repo
up-to-date on such a server, anyway. For non-bare repos on locally
accessible boxes, the [[pairing]] configurator will be the thing
to use, instead of this one.

Note: While the remote ssh configurator works great, and you could even have the
assistant running on multiple computers and use it to point them all at the
same repo on a server, the assistant does not yet support keeping such a
network topology in sync. That needs some of the ideas in [[cloud]] to
happen, so clients can somehow inform each other when there are changes.
Until that happens, the assistant polls only every 30 minutes, so it'll
keep in sync with a 30 minute delay.

---

This configurator can also set up encryped rsync special remotes. Currently
it always encrypts them, using the shared cipher mode of git-annex's
encryption. That avoids issues with gpg key generation and distribution,
and was easy to get working.

---

I feel I'm in a good place now WRT adding repository configurator wizards
to the webapp. This one took about 2.5 days, and involved laying some
groundwork that will be useful for other repository configurators. And it
was probably one of the more complex ones.

Now I should be able to crank out configurators for things like Amazon S3,
Bup, Rsync.net, etc fairly quickly. First, I need to do a beta release of
the assistant, and start getting feedback from my backers to prioritize
what to work on.


code:
Now finished building a special configurator for rsync.net. While
this is just a rsync remote to git-annex, there are some tricky bits to
setting up the ssh key using rsync.net's restricted shell. The configurator
automates that nicely. It took about 3 hours of work, and 49 lines of
rsync.net specific code to build this.

[[!img /assistant/rsync.net.png]]

Thanks to rsync.net who heard of my Kickstarter and gave me a really
nice free lifetime account. BTW guys, I wish your restricted shell
supported '&&' in between commands, and returned a nonzero exit status when
the command fails. This would make my error handling work better.

I've also reworked the repository management page. Nice to see those
configurators start to fill in!

[[!img /assistant/repositories.png]]


code:
* On OSX, install a launcher plist file, to run the assistant on login,
  and a `git-annex-webapp.command` file in the desktop. This is not tested
  yet.
* Made the webapp display alerts when the inotify/kqueue layer has a
  warning message.
* Handle any crashes of each of the 15 or so named threads by displaying
  an alert. (Of course, this should never happen.)


code:
Started reading about ZeroMQ with the hope that it could do some firewall
traversal thing, to connect mutually-unroutable nodes. Well, it could, but
it'd need a proxy to run on a server both can contact, and lots of
users won't have a server to run that on. The XMPP approach used by
dvcs-autosync is looking like the likeliest way for git-annex to handle
that use case.

However, ZeroMQ did point in promising directions to handle another use
case I need to support: Local [[pairing]]. In fairly short order, I got
ZeroMQ working over IP Multicast (PGM), with multiple publishers sending
messages that were all seen by multiple clients on the LAN (actually the
WAN; works over OpenVPN too). I had been thinking about using
Avahi/ZeroConf for discovery of systems to pair with, but ZeroMQ is rather
more portable and easy to work with.

Unfortunatly, I wasn't able to get ZeroMQ to behave reliably enough.
It seems to have some timeout issues the way I'm trying to use it,
or perhaps its haskell bindings are buggy? Anyway, it's really overkill
to use PGM when all I need for git-annex pairing discovery is lossy
UDP Multicast. Haskell has a simple `network-multicast` library for that,
and it works great.

With discovery out of the way (theoretically), the hard part about
[[pairing]] is going to be verifying that the desired repository is being
paired with, and not some imposter. My plan to deal with this involves a
shared secret, that can be communicated out of band, and HMAC. The webapp
will prompt both parties to enter the same agreed upon secret (which could
be any phrase, ideally with 64 bytes of entropy), and will then use it as
the key for HMAC on the ssh public key. The digest will be sent over the
wire, along with the ssh public key, and the other side can use the shared
secret to verifiy the key is correct.

The other hard part about [[pairing]] will be finding the best address to
use for git, etc to connect to the other host. If MDNS is available, it's
ideal, but if not the pair may have to rely on local DNS, or even
hard-coded IPs, which will be much less robust. Or, the assistant could
broadcast queries for a peer's current IP address itself, as a poor man's
MDNS.

All right then! That looks like a good week's worth of work to embark on.

---

Slight detour to package the haskell network-multicast library and upload
to Debian unstable.

Roughed out a data type that models the whole pairing conversation,
and can be serialized to implement it. And a state machine to run
that conversation. Not yet hooked up to any transport such as multicast
UDP.


code:
About half way done with implementing [[pairing]]. The webapp's interface
to prompt for a secret and start pairing is done; the protocol is
implemented; broadcasting of pairing requests is working; added Yet Another
Thread to listen for incoming pairing traffic.

Very happy with how this came together; starting with defining the protocol
with data types let me rapidly iterate until I had designed a simple, clean,
robust protocol. The implementation works well too; it's even possible to
start pairing, and only then bring up the network interface to the machine
you intended to pair with, and it'll detect the new interface and start
sending requests to it.

Next, I need to make alerts have a button that performs a stored
IO action. So that the incoming pair request alert can have a button to
respond to the pair request. And then I need to write the code to actually
perform the pairing, including ssh key setup.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmubB1Sj2rwFoVdZYvGV0ACaQUJQyiJXJI"
 nickname="Paul"
 subject="Hogging memory"
 date="2012-09-09T15:51:40Z"
 content="""
How do you prefer bugs to be reported?

I'm having a problem where git-annex (from git, 1e41c0d85ecc24e8656bff79b2fba46c3663a054) is taking over 20GB of RAM after adding a single file.

To reproduce:

1. Create a new annex (I used the web app, created an annex at ~/annex)
2. Add box.com as a remote, using encryption (followed <http://git-annex.branchable.com/tips/using_box.com_as_a_special_remote/>, basically `cd ~/annex; git annex initremote box.com type=directory directory=/media/box.com encryption=$GPGID`)
3. copied a file into the annex (ok, I admit, the file was a bit large, 350MB, but still)
4. refreshed the web app, nothing happened
5. closed the web app
6. started git annex assistant inside ~/annex
7. git annex webapp
8. noted that git annex noticed the file, and started transferring
9. wait
10. memory usage for the git-annex process goes beyond 21GB RAM
11. oomkiller kills git-annex
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.8.30"
 subject="comment 2"
 date="2012-09-09T16:25:52Z"
 content="""
The [[bugs]] page is the place to put bug reports like this so I won't forget them.

This should certainly not be happening. There are actually two git-annex processes running in the situation you describe; I'd be most curious to know whether the `git annex transfer` process was the one that blew up, or if the `git annex assistant` blew up. Also, it's not clear to me if you enabled the chunksize parameter when setting up the special remote, which could well be a significant detail.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmubB1Sj2rwFoVdZYvGV0ACaQUJQyiJXJI"
 nickname="Paul"
 subject="comment 3"
 date="2012-09-09T17:34:07Z"
 content="""
I'm almost certain that I specified chunksize like in the tip. How do I check?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.8.30"
 subject="comment 4"
 date="2012-09-09T18:00:48Z"
 content="""
You can see the full configuration with: git show git-annex:remote.log

(You probably do not want to paste the cipher= part of that here, although it *is* encrypted with your gpg key, unless you are using the shared encryption mode.)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmubB1Sj2rwFoVdZYvGV0ACaQUJQyiJXJI"
 nickname="Paul"
 subject="comment 5"
 date="2012-09-10T14:54:17Z"
 content="""
chunksize=2m was defined. I tried again, and the transfers again didn't seem to start. Then I did `killall git-annex`, cleaned the files from the box.com account, and tried again. This time it seems to be working, but the webapp isn't showing progress. Memory usage also stays reasonable, and doesn't seem to grow. Maybe it was just bad handling of unexpected repository state?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.8.30"
 subject="comment 6"
 date="2012-09-10T17:31:20Z"
 content="""
Re transfer progress not being shown, that's expected as upload progress displays are not yet implemented in the webapp.

I can't think of a way git-annex would care what was in your box.com repo. It just makes directories as needed, and will overwrite existing files in the rare case they already exist. And ignore any non-git-annex files.

Are you sure it was git-annex's memory use blowing up, and not the memory use of the davfs2 daemon?
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmubB1Sj2rwFoVdZYvGV0ACaQUJQyiJXJI"
 nickname="Paul"
 subject="comment 7"
 date="2012-09-10T17:35:48Z"
 content="""
Positive. I was grepping for git-annex from ps for the mem usage.
"""]]


code:
Alerts can now have buttons, that go to some url when clicked. Yay.

Implementing that was a PITA, because Yesod really only wants its type-safe
urls to be rendered from within its Handler monad. Which most things that
create alerts are not. I managed to work around Yesod's insistence on this
only by using a MVar to store the pure function that Yesod uses internally.
That function can only be obtained once the webapp is running.

----

Fixed a nasty bug where using gpg would cause hangs. I introduced this back
when I was reworking all the code in git-annex that runs processes, so it
would work with threading. In the process a place that had forked a process
to feed input to gpg was lost. Fixed it by spawning a thread to feed gpg.
Luckily I have never released a version of git-annex with that bug, but
the many users who are building from the master branch should update.

----

Made alerts be displayed while pairing is going on, with buttons to cancel
pairing or respond to a pairing request.


code:
Worked on [[pairing]] all day. It's complicated and I was close to being in
the weeds at times. I think it probably works now, but I have not tested it
at all. Tomorrow, testing, and cleaning up known problems.

----

Also ordered 1.5 terabytes of USB keys and a thousand git-annex stickers
today.


code:
Tons of pairing work, which culminated today in pairing fully working for
the very first time. And it works great! Type something like "my
hovercraft is full of eels" into two git annex webapps on the same LAN
and the two will find each other, automatically set up ssh keys, and sync
up, like magic. Magic based on math.

* Revert changes made to `authorized_keys` when the user cancels
  a pairing response. Which could happen if the machine that sent the
  pairing request originally is no longer on the network.
* Some fixes to handle lossy UDP better. Particularly tricky at the end
  of the conversation -- how do both sides reliably know when a
  conversation is over when it's over a lossy wire? My solution is just
  to remember some conversatons we think are over, and keep saying
  "this conversation is over" if we see messages in that conversation.
  Works.
* Added a UUID that must be the same in related pairing messages.
  This has a nice security feature: It allows detection of brute-force
  attacks to guess the shared secret, after the first wrong guess!
  In which case the pairing is canceled and a warning printed.
* That led to a thorough security overview, which I've added to
  the [[pairing]] page. Added some guards against unusual attacks,
  like console poisioning attacks. I feel happy with the security
  of pairing now, with the caveats that only I have reviewed it (and
  reviewing your own security designs is never ideal), and that the
  out-of-band shared secret communication between users is only as good
  as they make it.
* Found [a bug](https://github.com/yesodweb/yesod/issues/421)
  in Yesod's type safe urls. At least, I think it's a bug. Worked around it.
* Got very stuck trying to close the sockets that are opened to send
  multicast pairing messages. Nothing works, down to and including calling
  C `close()`. At the moment I have a socket leak. :(
  I need to understand the details of multicast sockets better to fix this.
  Emailed the author of the library I'm using for help.


code:
Kickstarter is over. Yay!

Today I worked on the bug where `git annex watch` turned regular files
that were already checked into git into symlinks. So I made it check
if a file is already in git before trying to add it to the annex.

The tricky part was doing this check quickly. Unless I want to write my
own git index parser (or use one from Hackage), this check requires running
`git ls-files`, once per file to be added. That won't fly if a huge
tree of files is being moved or unpacked into the watched directory.

Instead, I made it only do the check during `git annex watch`'s initial
scan of the tree. This should be OK, because once it's running, you
won't be adding new  files to git anyway, since it'll automatically annex
new files. This is good enough for now, but there are at least two problems
with it:

* Someone might `git merge` in a branch that has some regular files,
  and it would add the merged in files to the annex.
* Once `git annex watch` is running, if you modify a file that was
  checked into git as a regular file, the new version will be added
  to the annex.

I'll probably come back to this issue, and may well find myself directly
querying git's index.

---

I've started work to fix the memory leak I see when running `git annex
watch` in a large repository (40 thousand files). As always with a Haskell
memory leak, I crack open [Real World Haskell's chapter on profiling](http://book.realworldhaskell.org/read/profiling-and-optimization.html).

Eventually this yields a nice graph of the problem:

[[!img profile.png alt="memory profile"]]

So, looks like a few minor memory leaks, and one huge leak. Stared
at this for a while and trying a few things, and got a much better result:

[[!img profile2.png alt="memory profile"]]

I may come back later and try to improve this further, but it's not bad memory
usage. But, it's still rather slow to start up in such a large repository,
and its initial scan is still doing too much work. I need to optimize
more..


code:
I've changed the default backend used by git-annex from SHA256 to SHA256E.
Including the filename extension in the key is known to make repositories
more usable on things like MP3 players, and I've recently learned it also
avoids [[forum/Weird_behavior_with_OS_X_Finder_and_Preview.app]].

I thought about only changing the default in repositories set up by the
assistant, but it seemed simpler to change the main default. The old
backend is really only better if you might have multiple copies of files
with the same content that have different extensions.

Fixed the socket leak in pairing that eluded me earlier.

I've made a new [[polls]] page, and posted a poll:
[[polls/prioritizing_special_remotes]]


code:
It's possible for one git annex repository to configure a special remote
that it makes sense for other repositories to also be able to use. Today I
added the UI to support that; in the list of repositories, such
repositories have a "enable" link.

To enable pre-existing rsync special remotes, the webapp has to do the same
probing and ssh key setup that it does when initially creating them.
Rsync.net is also handled as a special case in that code. There was one
ugly part to this.. When a rsync remote is configured in the webapp,
it uses a mangled hostname like "git-annex-example.com-user", to
make ssh use the key it sets up. That gets stored in the `remote.log`, and so
the enabling code has to unmangle it to get back to the real hostname.

---

Based on the still-running [[prioritizing_special_remotes]] poll, a lot
of people want special remote support for their phone or mp3 player.
(As opposed to running git-annex on an Android phone, which comes later..)
It'd be easy enough to make the webapp set up a directory special remote
on such a device, but that makes consuming some types of content on the
phone difficult (mp3 players seem to handle them ok based on what people tell
me). I need to think more about some of the ideas mentioned in [[android]]
for more suitable ways of storing files.

One thing's for sure: You won't want the assistant to sync all your files
to your phone! So I also need to start coming up with partial syncing
controls. One idea is for each remote to have a configurable matcher for files
it likes to receive. That could be only mp3 files, or all files inside a
given subdirectory, or all files *not* in a given subdirectory. That means
that when the assistant detects a file has been moved, it'll need to add
(or remove) a queued transfer. Lots of other things could be matched on,
like file size, number of copies, etc. Oh look, I have a
[beautiful library I wrote earlier](http://joeyh.name/blog/entry/happy_haskell_hacker)
that I can reuse!


code:
Started today doing testing of [[syncing]], and found some bugs and things
it needs to do better. But was quickly sidetracked when I noticed that
`transferkey` was making a commit to the git-annex branch for every file it
transferred, which is too slow and bloats history too much.

To fix that actually involved fixing a long-standing annoyance; that
read-only git-annex commands like `whereis` sometimes start off with
"(Recording state in git)", when the journal contains some not yet
committed changes to the git-annex branch. I had to carefully think
through the cases to avoid those commits.

As I was working on that, I found a real nasty lurking bug in the git-annex
branch handling. It's unlikely to happen unless `annex.autocommit=false` is
set, but it could occur when two git-annex processes race one another just
right too. The root of the bug is that `git cat-file --batch` does not
always show changes made to the index after it started. I think it does
in enough cases to have tricked me before, but in general it can't be
trusted to report the current state of the index, but only some past state.

I was able to fix the bug, by ensuring that changes being made to the
branch are always visible in either the journal or the branch -- never in
the index alone.

----

Hopefully something less low-level tomorrow..!


code:
Syncing works well when the graph of repositories is strongly connected.
Now I'm working on making it work reliably with less connected graphs.

I've been focusing on and testing a doubly-connected list of repositories,
such as: `A <-> B <-> C`

----

I was seeing a lot of git-annex branch push failures occuring in
this line of repositories topology. Sometimes was is able to recover from
these, but when two repositories were trying to push to one-another at the
same time, and both failed, both would pull and merge, which actually keeps
the git-annex branch still diverged. (The two merge commits differ.)

A large part of the problem was that it pushed directly into the git-annex
branch on the remote; the same branch the remote modifies. I changed it to
push to `synced/git-annex` on the remote, which avoids most push failures.
Only when A and C are both trying to push into `B/synced/git-annex` at the
same time would one fail, and need to pull, merge, and retry.

-----

With that change, git syncing always succeeded in my tests, and without
needing any retries. But with more complex sets of repositories, or more
traffic, it could still fail.

I want to avoid repeated retries, exponential backoffs, and that kind of
thing. It'd probably be good enough, but I'm not happy with it because
it could take arbitrarily long to get git in sync.

I've settled on letting it retry once to push to the synced/git-annex
and synced/master branches. If the retry fails, it enters a fallback mode,
which is guaranteed to succeed, as long as the remote is accessible.

The problem with the fallback mode is it uses really ugly branch names.
Which is why Joachim Breitner and I originally decided on making `git annex
sync` use the single `synced/master` branch, despite the potential for
failed syncs. But in the assistant, the requirements are different,
and I'm ok with the uglier names.

It does seem to make sense to only use the uglier names as a fallback,
rather than by default. This preserves compatability with `git annex sync`,
and it allows the assistant to delete fallback sync branches after it's
merged them, so the ugliness is temporary.

---

Also worked some today on a bug that prevents C from receiving files
added to A.

The problem is that file contents and git metadata sync independantly. So C
will probably receive the git metadata from B before B has finished
downloading the file from A. C would normally queue a download of the
content when it sees the file appear, but at this point it has nowhere to
get it from.

My first stab at this was a failure. I made each download of a file result
in uploads of the file being queued to every remote that doesn't have it
yet. So rather than C downloading from B, B uploads to C. Which works fine,
but then C sees this download from B has finished, and proceeds to try to
re-upload to B. Which rejects it, but notices that this download has
finished, so re-uploads it to C...

The problem with that approach is that I don't have an event when a download
succeeds, just an event when a download ends.  Of course, C could skip
uploading back to the same place it just downloaded from, but loops are
still possible with other network topologies (ie, if D is connected to both
B and C, there would be an upload loop 'B -> C -> D -> B`). So unless I can
find a better event to hook into, this idea is doomed.

I do have another idea to fix the same problem. C could certainly remember
that it saw a file and didn't know where to get the content from, and then
when it receives a git push of a git-annex branch, try again.


code:
Implemented deferred downloads. So my example from yesterday of three
repositories in a line keep fully in sync now!

I punted on one problem while doing it. It might be possible to get a really
big list of deferred downloads in some situation. That all lives in memory.
I aim for git-annex to always have a constant upper bound on memory use,
so that's not really acceptable. I have TODOed a reminder to do something
about limiting the size of this list.

----

I also ran into a nasty crash while implementing this, where two threads
were trying to do things to git HEAD at the same time, and so one crashed,
and in a way I don't entirely understand, that crash took down another
thread with a BlockedIndefinitelyOnSTM exception. I think I've fixed
this, but it's bothersome that this is the second time that modifications
to the Merger thread have led to a concurrency related crash that I
have not fully understood.

My guess is that STM can get confused when it's
retrying, and the thread that was preventing it from completing a
transaction crashes, because it suddenly does not see any other
references to the TVar(s) involved in the transaction. Any GHC STM gurus
out there?

---

Still work to be done on making data transfers to keep fully in sync in all
circumstances. One case I've realized needs work occurs when a USB drive is
plugged in. Files are downloaded from it to keep the repo in sync, but the
repo neglects to queue uploads of those files it just got out to other
repositories it's in contact with. Seems I still need to do something to
detecting when a successful download is done, and queue uploads.


code:
Turns out I was able to easily avoid the potential upload loops that would
occur if each time a repo receives a download, it queues uploads to the
repos it's connected to. With that done. I suspect, but have not proven,
that the assistant is able to keep repos arranged in any shape of graph in
sync, as long as it's connected (of course) and each connection is
bi-directional. That's a good start .. or at least a nice improvement from
only strongly connected graphs being kept in sync.

Eliminated some empty commits that would be made sometimes, which is a nice
optimisation.

------

I wanted to get back to some UI work after this week's deep dive into the
internals. So I filled in a missing piece, the repository switcher in the
upper right corner. Now the webapp's UI allows setting up different
repositories for different purposes, and switching between them.


code:
Putting together a shortlist of things I want to sort out before the beta.

* [[Progress bars|progressbars]] for file uploads.
* No mocked up parts in the webapp's UI. Think I implemented the last of
  those yesterday, although there are some unlinked repository configuration
  options.
* The basic watching functionality, should work reliably.
  There are some known scalability issues with eg,
  [[kqueue on OSX|bugs/Issue_on_OSX_with_some_system_limits]] that
  need to be dealt with, but needn't block a beta.
* Should keep any configuration of repositories that can be set up using
  the webapp in sync whenever it's possible to do so. I think that'll work
  after the past few days work.
* Should be easy to install and get running. Of course part of the point
  of the beta release is to get it out there, on Hackage, in Debian
  unstable, and in the other places that git-annex packagers put it.
  As to getting it running, the autostart files and menu items look good
  on Linux. The OSX equivilants still need work and testing.
* No howlingly bad bugs. [[This bug|bugs/pasting_into_annex_on_OSX]] is
  the one I'm most concerned with currently. OTOH,
  [[bugs/watcher_commits_unlocked_files]] can be listed in the errata.

----

So I worked on progress bars for uploads today. Wrote a nice little
parser for rsync's progress output, that parses arbitrary size chunks,
returning any unparsable part. Added a ProgressCallback parameter to
all the backends' upload methods. Wrote a nasty thing that intercepts
rsync's output, currently a character at a time (horrible, but rsync
doesn't output that much, so surprisingly acceptable), and outputs it and
parses it. Hooked all this up, and got it working for uploads to
git remotes. That's 1/10th of the total ways uploads can happen that
have working progress bars. It'll take a while to fill in the rest..


code:
Worked more on upload progress tracking. I'm fairly happy with its state
now:

* It's fully implemented for rsync special remotes.

* Git remotes also fully support it, with the
  notable exception of file uploads run by `git-annex-shell recvkey`. That
  runs `rsync --server --sender`, and in that mode, rsync refuses to output
  progress info. Not sure what to do about this case. Maybe I should
  write a parser for the rsync wire protocol that can tell what chunk of the
  file is being sent, and shim it in front of the rsync server? That's
  rather hardcore, but it seems the best of a bad grab bag of options that
  include things like `LD_PRELOAD` hacks.

* Also optimised the rsync progress bar reader to read whole
  chunks of data rather than one byte at a time.

* Also got progress bars to actually update in the webapp for uploads.

  This turned out to be tricky because kqueue cannot be used to detect when
  existing files have been modified. (One of kqueue's worst shortcomings vs
  inotify.) Currently on kqueue systems it has to poll.

I will probably upload add progress tracking to the directory special remote,
which should be very easy (it already implements its own progress bars),
and leave the other special remotes for later. I can add upload progress
tracking to each special remote when I add support for configuring it in
the webapp.


code:
Short day today, but I again worked only on progress bars.

* Added upload progress tracking for the directory special remote.
* Some optimisations.
* Added a `git annex-shell transferkey` command. This isn't used yet,
  but the plan is to use it to feed back information about how much
  of a file has been sent when downloading it. So that the uploader
  can display a progress bar. This method avoids needing to parse the rsync
  protocol, which is approximately impossible without copying half of rsync.
  Happily, git-annex's automatic ssh connection caching will make the small
  amount of data this needs to send be efficiently pipelined over the same
  ssh connection that rsync is using.

I probably have less than 10 lines of code to write to finish up
[[progressbars]] for now. Looking forward to getting that behind me, and on
to something more interesting. Even doing mail merge to print labels to
mail out Kickstarter rewards is more interesting than progress bars at this
point. :)


code:
Finally wrapped up progress bars; upload progress is now reported in all
situations.

After all that, I was pleased to find a use for the progress info, beyond
displaying it to the user. Now the assistant uses it to decide whether it
makes sense to immediately retry a failed transfer. This should make it
work nicely, or at least better, with flaky network or drives.

The webapp crashed on startup when there was no `~/.gitconfig`.
Guess all of us who have tried it so far are actual git users,
but I'm glad I caught this before releasing the beta.

Jimmy Tang kindly took on making a OS X .app directory for git-annex.
So it now has an icon that will launch the webapp.
[[!img /assistant/osx-app.png]]

I'm getting lots of contributors to git-annex all of a sudden. I've had 3
patches this weekend, and 2 of them have been to Haskell code.
Justin Azoff is working on [[todo/incremental_fsck]], and Robie Basak
has [gotten Amazon Glacier working](https://github.com/basak/glacier-cli)
using the hook special remote.

Started doing some design for [[transfer_control]]. I will start
work on this after releasing the first beta.


code:
Since last post, I've worked on speeding up `git annex watch`'s startup time
in a large repository.

The problem was that its initial scan was naively staging every symlink in
the repository, even though most of them are, presumably, staged correctly
already. This was done in case the user copied or moved some symlinks
around while `git annex watch` was not running -- we want to notice and
commit such changes at startup.

Since I already had the `stat` info for the symlink, it can look at the
`ctime` to see if the symlink was made recently, and only stage it if so.
This sped up startup in my big repo from longer than I cared to wait (10+
minutes, or half an hour while profiling) to a minute or so. Of course,
inotify events are already serviced during startup, so making it scan
quickly is really only important so people don't think it's a resource hog.
First impressions are important. :)

But what does "made recently" mean exactly? Well, my answer is possibly
over engineered, but most of it is really groundwork for things I'll need
later anyway. I added a new data structure for tracking the status of the
daemon, which is periodically written to disk by another thread (thread #6!)
to `.git/annex/daemon.status` Currently it looks like this; I anticipate
adding lots more info as I move into the [[syncing]] stage:

	lastRunning:1339610482.47928s
	scanComplete:True

So, only symlinks made after the daemon was last running need to be
expensively staged on startup. Although, as RichiH pointed out,
this fails if the clock is changed. But I have been planning to have a
cleanup thread anyway, that will handle this, and other
potential problems, so I think that's ok.

Stracing its startup scan, it's fairly tight now. There are some repeated
`getcwd` syscalls that could be optimised out for a minor speedup.

----

Added the sanity check thread. Thread #7! It currently only does one sanity
check per day, but the sanity check is a fairly lightweight job,
so I may make it run more frequently. OTOH, it may never ever find a
problem, so once per day seems a good compromise.

Currently it's only checking that all files in the tree are properly staged
in git. I might make it `git annex fsck` later, but fscking the whole tree
once per day is a bit much. Perhaps it should only fsck a few files per
day? TBD

Currently any problems found in the sanity check are just fixed and logged.
It would be good to do something about getting problems that might indicate
bugs fed back to me, in a privacy-respecting way. TBD

----

I also refactored the code, which was getting far too large to all be in
one module.

I have been thinking about renaming `git annex watch` to `git annex assistant`,
but I think I'll leave the command name as-is. Some users might
want a simple watcher and stager, without the assistant's other features
like syncing and the webapp. So the next stage of the
[[roadmap|design/assistant]] will be a different command that also runs
`watch`.

At this point, I feel I'm done with the first phase of [[inotify]].
It has a couple known bugs, but it's ready for brave beta testers to try.
I trust it enough to be running it on my live data.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="Battery usage"
 date="2012-06-15T09:57:33Z"
 content="""
Complete fsck is good, but once a week probably enough.

But please see if you can make fsck optional depending on if the machine is running on battery.
"""]]


code:
Just released git-annex 3.20120924, which includes beta versions of
the assistant and webapp. Read the [[/assistant/errata]], then give it a
try!

I've uploaded it to Haskell's cabal, and to Debian unstable, and hope my
helpers for other distributions will update them soon. (Although the
additional dependencies to build the webapp may take a while on some.)
I also hope something can be done to make a prebuilt version available on
OSX soonish.

I've decided to license the webapp under the
[AGPL](http://www.gnu.org/licenses/agpl-3.0.html). This should not impact
normal users of it, and git-annex can be built without the webapp as a pure
GPL licensed program. This is just insurance to prevent someone turning the
webapp into a propritary web-only service, by requiring that anyone who
does so provide the source of the webapp.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlJEI45rGczFAnuM7gRSj4C6s9AS9yPZDc"
 nickname="Kevin"
 subject="Haskell on MacPorts"
 date="2012-09-24T21:09:39Z"
 content="""
FYI, I just came across a [MacPorts ticket](https://trac.macports.org/ticket/35813) to update Haskell to 2012.2.0.0.  It would be nice to install git-annex from MacPorts.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 2"
 date="2012-09-24T21:21:57Z"
 content="""
cool, I haven't been on top of things recently. If haskell-platform gets updated in macports and assuming its easy to package up any dependancies (if any) as macport packages, git-annex should be relatively straightforward to package up
"""]]