code:
Slowly getting through the bugs that were opened while I was on vacation and
then I'll try to get to all the comments. 60+ messages to go.

Got git-annex working better on encfs, which does not support hard links in
paranoid mode. Now git-annex can be used in indirect mode, it doesn't force
direct mode when hard links are not supported.

Made the Android repository setup special case generate a .gitignore file
to ignore thumbnails. Which will only start working once the assistant
gets .gitignore support.

-----

Been thinking today about encrypting XMPP traffic, particularly git push
data. Of course, XMPP is already encrypted, but that doesn't hide it from
those entities who have access to the XMPP server or its encryption key.
So adding client-to-client encryption has been on the TODO list all along.

OTR would be a great way to do it. But I worry that the confirmation steps
OTR uses to authenticate the recipient would make the XMPP pairing UI harder
to get through.

Particularly when pairing your own devices over XMPP, with several devices
involved, you'd need to do a lot of cross-confirmations. It would be better
there, I think, to just use a shared secret for authentication. (The need
to enter such a secret on each of your devices before pairing them would
also provide a way to use different repositories with the same XMPP
account, so 2birds1stone.)

Maybe OTR confirmations would be ok when setting up sharing with a friend.
If OTR was not used there, and it just did a Diffie-Hellman key exchange
during the pairing process, it could be attacked by an active MITM spoofing
attack. The attacker would then know the keys, and could decrypt future
pushes. How likely is such an attack? This goes far beyond what we're
hearing about. Might be best to put in some basic encryption now, so
we don't have to worry about pushes being passively recorded on the
server. Comments appreciated.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlJ2utMQgMEYAOs3Dfc6eZRyUzt4acNXUU"
 nickname="David"
 subject="comment 1"
 date="2013-06-10T23:42:20Z"
 content="""
If you do a D-H key exchange and display the key fingerprint on both devices, that would be a pretty strong disincentive against XMPP servers doing a MitM. You only need one victim to casually notice a discrepancy, and it leaves behind a strong proof of tampering.
"""]]

repo_name: kakafarm/git

license: unknown


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 nickname="joey"
 subject="comment 2"
 date="2013-06-10T23:50:27Z"
 content="""
David, along those lines, if a value derived from the D-H key were committed to the git repo, then the MITM would need to maintain two distinct git trees, and translate between them on an ongoing basis. Which is harder. And then if both clients at any point communicated via another channel (ie, local pairing), the forgery would be very noticeable.

Which is not to say that this is necessarily good enough..
"""]]


code:
Got caught up on my backlog yesterday.

Part of adding files in direct mode involved removing write permission from
them temporarily. That turned out to cause problems with some programs that
open a file repeatedly, and was generally against the principle that direct
mode files are always directly available. Happily, I was able to get rid of
that without sacrificing any safety.

Improved syncing to bare repositories. Normally syncing pushes to a
synced/master branch, which is good for non-bare repositories since git
does not allow pushing to the currently checked out branch. But for bare
repositories, this could leave them without a master branch, so cloning
from them wouldn't work. A funny thing is that git does not really have any
way to tell if a remote repository is bare or not. Anyway, I did put in a
fix, at the expense of pushing twice (but the git data should only be
transferred once anyway).


code:
Today I got to deal with bugs on Android (busted use of `cp` among other
problems), Windows (fixed a strange hang when adding several files), and
Linux (`.desktop` files suck and Wine ships a particularly nasty one).
Pretty diverse!

Did quite a lot of research and thinking on XMPP encryption yesterday, but
have not run any code yet (except for trying out a D-H exchange in `ghci`).
I have listed several options on the [[XMPP]] page.

Planning to take a look at
[[bugs/Handling_of_files_inside_and_outside_archive_directory_at_the_same_time]]
tomorrow; maybe I can come up with a workaround to avoid it behaving so
badly in that case.


code:
Yay, I fixed the
[[bugs/Handling_of_files_inside_and_outside_archive_directory_at_the_same_time]]
bug!
At least in direct mode, which thanks to its associated files tracking
knows when a given file has another file in the repository with the same
content. Had not realized the behavior in direct mode was so bad, or the
fix so relatively easy. Pity I can't do the same for indirect mode, but
the problem is much less serious there.

That was this weekend. Today, I nearly put out a new release (been 2 weeks
since the last one..), but ran out of time in the end, and need to get the
OSX autobuilder fixed first, so have deferred it until Friday.

However, I did make some improvements today.
Added an `annex.debug` git config setting, so debugging can
be turned on persistently. People seem to expect that to happen when
checking the checkbox in the webapp, so now it does.

Fixed 3 or 4 bugs in the Windows port. Which actually, has users now, or
at least one user. It's very handy to actually get real world testing of
that port.

[[!meta date="Mon, 17 Jun 2013 17:14:04 -0400"]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 1"
 date="2013-06-26T08:36:33Z"
 content="""
Been pretty bad with keeping the osx builder going, but this should fix it...

<pre>
[jtang@x00 build ((cfb577d...))]$ git diff
diff --git a/.gitignore b/.gitignore
index 93bd49e..717b58a 100644
--- a/.gitignore
+++ b/.gitignore
@@ -24,3 +24,5 @@ cabal-dev
 .virthualenv
 tags
 Setup
+*.hi
+*.o
</pre>

The build on OSX leaves some stray files and it complains about it.
"""]]


code:
One of my Windows fixes yesterday got the test suite close to sort of
working on Windows, and I spent all day today pounding on it. Fixed
numerous bugs, and worked around some weird Windows behaviors -- like
recursively deleting a directory sometimes fails with a permission denied
error about a file in it, and leaves behind an empty directory. (What!?)
The most important bug I fixed caused CR to leak into files in the
git-annex branch from Windows, during a union merge, which was not a good
thing at all.

At the end of the day, I only have 6 remaining failing test cases on
Windows. Half of them are some problem where running `git annex sync`
from the test suite stomps on PATH somehow and prevents xargs from working.
The rest are probably real bugs in the directory (again something to do
with recursive directory deletion, hmmm..), hook, and rsync
special remotes on Windows. I'm punting on those 6 for now, they'll be
skipped on Windows.

Should be worth today's pain to know in the future when I break
something that I've oh-so-painfully gotten working on Windows.


code:
Pushed out a release today. While I've somewhat ramped down activity this
month with the Kickstarter period over and summer trips and events ongoing,
looking over the changelog I still see a ton of improvements in the 20 days
since the last release.

Been doing some work to make the assistant daemon be more `nice`. I don't
want to nice the whole program, because that could make the web interface
unresponsive. What I am able to do, thanks to Linux violating POSIX, is to
`nice` certain expensive operations, including the startup scan and the daily
sanity check. Also, I put in a call to `ionice` (when it's available)
when `git annex assistant --autostart` is run, so the daemon's
disk IO will be prioritized below other IO. Hope this keeps it out of your
way while it does its job.


code:
Got caught up on a big queue of messages today. Mostly I hear from people
when git-annex is not working for them, or they have a question about using
it. From time to time someone does mention that it's working for them.

> We have 4 or so machines all synching with each other via the local
> network thing. I'm always amazed when it doesn't just explode :)

Due to the nature of git-annex, a lot of people can be using it without
anyone knowing about it. Which is great. But these little success stories
can make all the difference. It motivates me to keep pounding out the
development hours, it encourages other people to try it, and it'd be a good
thing to be able to point at if I tried to raise more funding now that I'm
out of Kickstarter money.

I'm posting my own success story to my main blog:
[git annex and my mom](http://joeyh.name/blog/entry/git_annex_and_my_mom)

If you have a success story to share, why not blog about it, microblog it,
or just post a comment here, or even send me a private message. Just a
quick note is fine. Thanks!

----

Going through the bug reports and questions today, I ended up fixing
three separate bugs that could break setting up a repo on a remote ssh
server from the webapp.

Also developed a minimal test case for some gnucash behavior that prevents
the watcher from seeing files it writes out. I understand the problem,
but don't have a fix for that yet. Will have to think about it. (A year ago
today, my blog featured the
[[first_release_of_the_watcher|day_16__more_robust_syncing]].)


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmnG4EuvZWse5hvgrl0XAK-U61e-0iGaao"
 nickname="David"
 subject="Nothing fancy but..."
 date="2013-06-27T16:52:12Z"
 content="""
Hi Joey,

First, thanks very much for your effort put into git-annex.  I store my photos in git-annex and I like it because:

* I want to store multiple copies of the pictures and I can see where they are,
* I can have a global view of the collection on my laptop without carrying all the files.
* the indirect mode makes it difficult to me to do stupid things.

So there's nothing fancy, but it works for me very well.  From time to time I try the assistant's new features and I'm planning to replace Dropbox eventually.

Secondly, thanks very much for the dev blog!  I was looking forward every morning to see what's coming next, what problems you are facing and how you're planning to solve them.  It is simply amazing what you've achieved in the last year and your posts gave me a lot of motivation too (in finishing my thesis  :-))!

Thanks!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.193"
 subject="comment 11"
 date="2013-06-27T18:11:03Z"
 content="""
Excerpt from a [longer comment](http://git-annex.branchable.com/bugs/Problems_with_syncing_gnucash/#comment-7d70d822a0fdf672ba09535f943af7c9) by Florian:

I know that my expectations in this project are a little bit high and I see how much work it was to get where we are now. Once again thank you for the excellent work done so far. I don't fear to synchronize important data with git-annex. I even use git-annex to backup and synchronize the member database for our club (wannabe hackerspace).
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.193"
 subject="comment 12"
 date="2013-06-27T18:11:45Z"
 content="""
From a mail I received: --[[Joey]]

Git-annex makes it possible for me to have lots of data.  Before I
started using it I had a music library and a few massive tar files,
none of it properly backed up, and occasionally there would be random
corruption (I still have broken mp3s which jump and stutter in places)
because I would carelessly cp them from HDD to HDD.  So I'd avoid
actually letting data accumulate; this was inconvenient in itself.
Now git-annex does everything for me.

Following your development blog, as a novice programmer, has been
really enjoyable and interesting.  My only experience of professional
IT has put me of but it's good to see you working through things
methodically and properly.
"""]]


code:
[[!comment format=mdwn
 username="http://julien.lefrique.name/"
 nickname="jlefrique"
 subject="Many thanks!"
 date="2013-06-27T19:59:47Z"
 content="""
Hi Joey,

More than ten years ago, I was looking for a software to index the content of all my
CD-ROMs in order to create a kind of \"table of content\". Today there is
git-annex!

I discovered git-annex few months ago and it really helps me to keep my file
well organized. I missed the first crowd funding campaign but would be happy
to contribute to the next one :D

I mainly use four different annexes for:

* my private photos and videos,
* my music collection,
* my other multimedia files (movies...),
* my other files that are too large to be commited into git directly (pdf, binaries...).

I use git-annex manually via the command line. I started to play with the
assistant on my Android tablet and I plan to install it on my wife's
Macbook to synchronize her data with the NAS.

A standalone build for arm to run git-annex on my NAS (QNAP 219 PII) would be perfect :-P

Thank you very much for all your great softwares (yes I also use etckeeper,
some moreutils tools and more recently ikiwiki).

-Julien

"""]]


code:
[[!comment format=mdwn
 username="EvanDeaubl"
 ip="67.128.198.190"
 subject="comment 14"
 date="2013-06-28T18:35:55Z"
 content="""
git-annex has been working brilliantly for me. I have been slowly moving content out of the various places I've had it stored (file servers, folders hidden away on all my machines, Dropbox, Google Drive, etc.) into a single git-annex repo. The idea has been to mimic Dropbox's single folder, but with much more flexibility: I sync my working machines, my home fileserver, my VPS (for file access on the go), Android devices, and even a couple of dumb MP3 players from git-annex. Best of all, I have one place to go for all of my files, and if they're not on the device at the moment I need them, they're a download away. I didn't find git-annex until after the first crowdfunding campaign had completed, but if there's another, I would be a happy supporter.

"""]]


code:
[[!comment format=mdwn
 username="http://jasonwoof.com/"
 nickname="JasonWoof"
 subject="It's been great!"
 date="2013-06-29T03:48:08Z"
 content="""
Looks like I started using git-annex in November 2010.

I'd already been tracking lots of small files in my home directory (settings, scripts, notes, todo list, timesheets, invoices, etc) with git.

So awesome to get backups, syncing and manual merge conflict resolution.

git-annex extended that to handle my larger files. git-annex's ability to be selective about what is stored where, allowed my to still be able to git clone my home directory onto small devices (I've got a debian chroot with only 6GB or so, and I installed debian on my new chromebook, that's only got 16GB of disk space (for the whole system.))

I love having the full catalog visible on my filesystem with symlinks, and I can just request a file, and git-annex will figure out where to get it from, even if I'm away from home on my laptop.

I just got a 2nd external hard drive, and I'm delighted with how easy it was to get a backup of everything on there.

I was able to quickly and safely clear up space on my desktop computer! I'm confident that my data is safe because I've got git-annex configured to make sure there's at least two copies of my videos, and 3 copies of everything else.

I've finally gone from either keeping hard drives from my past computers or keeping a full copy of the data, to having a proper archiving and backup system for all files I care about.

I've got over 12,000 files stored on my main annex, and it's perfectly responsive.

Joey does a great job, and has been very responsive on the rare occasions where I needed help.

Three cheers!

--
Jason
"""]]


code:
[[!comment format=mdwn
 username="https://dzsino.myopenid.com/"
 nickname="dzsino"
 subject="thanks!"
 date="2013-06-30T13:10:35Z"
 content="""
I'm using git-annex to consolidate all the stuff that accumulated for years on various drives I have. Music I downloaded and bought (don't wanna loose those), photos from 2003 and on, archives of school and work stuff etc. Using it for 6-7 months now, I came to trust git-annex, I very much like the command line robustness it has and I really like that all my files are accessible in a single tree. It would be cool to have some metadata storage (for file dates, tags and such) and maybe spotlight-like search (elastic-search comes to mind)..
Still, without these, it has a great problem-solution fit for me!

Thanks Joey for all the great work, I really hope you can continue improving it, I would gladly join to a second kickstarter funding!

"""]]


code:
[[!comment format=mdwn
 username="GLITTAH"
 ip="50.23.191.90"
 subject="Several annexes managing >3TB. No problems!"
 date="2013-07-02T00:12:27Z"
 content="""
I've got an annex with all my DVD ISOs that's nearly 3TB alone, uses the SHA512 hash and gets split over three computers, 6 external SATA drives and 3 USB drives.  No errors so far, and the issues with git-annex I've had have all been addressed very quickly (thanks Joey!) and in a non-destructive manner so I don't have to re-init a repo.  I also use it to manage my music library, which allows to me to sleep at night without worrying about a corrupted file spreading through all my backups over time since git-annex has a fsck and hashing utility.  I can't imagine using anything else for managing collections of files: it offloads the task of keeping track of, getting, verifying, and general management of files so I don't even need to worry about the boring stuff, I just get around to watching/listening to my media collection.

Thanks for all the work, it's changed my workflow and file management (I get more sleep too)!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.193"
 subject="comment 18"
 date="2013-07-03T15:51:48Z"
 content="""
Another good tweet: --[[Joey]]

My google Reader replacement: newsbeuter with the data directory stored in an autosynced git-annex repo.

<https://twitter.com/JustinAzoff/status/352256199070199809>
"""]]


code:
[[!comment format=mdwn
 username="http://edheil.wordpress.com/"
 ip="99.54.57.201"
 subject="comment 1"
 date="2013-06-25T22:26:27Z"
 content="""
I've been real quiet cause everything's been working great. :)

One thing I have not messed with yet is the whole XMPP thing.  I mostly don't share between two machines that are online at the same time, I just use git annex for backups and to have content \"optionally\" available.  I should give XMPP a try sometime.

"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmhfodZquCI_EEl-f3h7HkROTszlsQL6yA"
 nickname="Joe"
 subject="comment 20"
 date="2013-07-04T12:41:35Z"
 content="""
I'm using git-annex to make videos, pictures and music available on all my devices (windows, android, linux). It has saved me on trips with my 2yr daughter when she's had meltdowns; I have her favorite movie ready. I quickly exceeded the free dropbox quota and git-annex has replaced the need and gives me greater confidence and control over my data. Thank you!
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm0sGxsiJ7yj5iQsF-A5tEl6XKOGQieqEo"
 nickname="Matthew"
 subject="Writing papers"
 date="2013-07-04T15:32:24Z"
 content="""
I'm a graduate student, so I write a lot.  I started using git to manage my latex sources a while ago, but I didn't like gumming up my repository with a bunch of figures (large binaries). I was constantly forgetting to scp the latest versions of my figures from here to there (as I do my work on a number of different computers.)  Git annex gives me a good way to version the figures and manage them with the same tool I'm using for the text.

My use case seems pretty far from typical, judging from the other comments here, so I thought I should add my two cents!  I'm constantly trying to sell this approach to my fellow students.

Also, I'm going to slip in a plug for another great Haskell project here -- Pandoc. I work with people who suggest changes to my drafts, but only using Word's \"Track Changes\".  So now I'm generating Word docs from Pandoc markdown for my collaborators to edit.  When it comes time to publish, I can convert my markdown to latex and clean up the formatting.

Why are all the really cool projects written in Haskell? :)
"""]]


code:
[[!comment format=mdwn
 username="http://sunny256.sunbase.org/"
 nickname="sunny256"
 subject="git-annex is teh awesome"
 date="2013-07-15T12:37:54Z"
 content="""
Just like Git, git-annex is teh awesome. I've used it for almost a year now, and I've never had more control of all my stuff. I store all my valuable (and also not so valuable) files in it, more than a terabyte, and it's rapidly growing. Movies, pictures, recordings, software, podcasts, DVDs and CDs, you name it. It's rock solid with a really good design. Joey says it isn't a backup solution, but I strongly disagree. My data have never been safer. git-annex ensures that I have several copies in different locations, I always know where all the copies are, and the disk space is used way more efficiently now that I don't have unnecessary duplicates everywhere anymore.
"""]]


code:
[[!comment format=mdwn
 username="https://launchpad.net/~subito"
 nickname="subito"
 subject="Big success!"
 date="2013-06-26T07:00:15Z"
 content="""
Annex is working great for me since 8/2012 or something... Using it a lot and the Assistant is awesome!

If you would raise some more funding, I'd be more than happy to support you once more - really great software!
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl3A5oeZFLreGhDTFVMOJYIy1auKiTL_ZY"
 nickname="Maggie"
 subject=":)"
 date="2013-06-26T15:04:45Z"
 content="""
Poets aren't humble people.
We revel in our own narcissism.
So being mentioned in my brother's blog
was great for me.

I am proud of the success of Git Annex.

(My poems tend to be _much_ better than this.)
"""]]


code:
[[!comment format=mdwn
 username="http://nicolas-schodet.myopenid.com/"
 ip="2a01:e35:8ae6:f130:1e4b:d6ff:fe78:1ddb"
 subject="Thank you for this great tool!"
 date="2013-06-26T18:10:05Z"
 content="""
I use git-annex for my picture collection. I can add pictures from my computers, or my FTP server when someone wants to send me their pictures. Then I push them to my online gallery. Git annex is great to keep all my repository synchronized while not requesting that all my pictures are present on my laptop.  Thank you!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.193"
 subject="comment 5"
 date="2013-06-26T19:39:15Z"
 content="""
Excerpt from an email I received: --[[Joey]]

Another story, but without the assistant this time: I've \"designed\" a
small f2f file sharing network with it, I'm using it with a friend to
share accross several computer. This is basically a workflow
with git annex and gitoline, the flexibility of git and the capacities
of git annex just gave us this out of the box. Some specs are here if
you are curious
http://worlddomination.be/ideas/2012/idea-n-21-build-a-f2f-sharing-network-using
-git-annex.html

"""]]


code:
[[!comment format=mdwn
 username="andy"
 ip="99.127.140.99"
 subject="Another success story"
 date="2013-06-27T02:03:00Z"
 content="""
I'm using Ubuntu, and I do a clean install about every 6 months when the new versions come out. I also have 2-3 systems I need to keep in sync. I use git annex to store files I want to keep between OS reinstalls, and to sync the various systems I need data on. I also use it to backup data to the cloud and to bup. I have 6+ repos, 235 GB of data, and 31,029 known annex keys (thanks `git annex map` and `git annex status`!); nevertheless git annex is generally pretty cooperative. I'm not using the assistant yet, but I'd like to eventually. I almost don't have to think about the stuff annex handles. It's really fun to install a fresh OS, clone the git annex repo, and have my entire annex directory tree available immediately.

Thanks Joey! You've done a great job!
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.108.193"
 subject="comment 7"
 date="2013-06-27T06:31:29Z"
 content="""
Amazing tweet I found! --[[Joey]]

Git-annex now auto-syncing photos from my android phone to a Tor hidden SSH service I control (via @guardianproject's Orbot) #prismbreak

<https://twitter.com/graphiclunarkid/status/347658443479449601>
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnAvbXOnK57sqgvZvxkbG74NUKBDwKDcuk"
 nickname="Tim"
 subject="Backups"
 date="2013-06-27T13:14:43Z"
 content="""
We use rsync and tar to backup to a server on the local network, and since we discoverd git-annex, we use this mechanism to automatically put all this data in both a local bup repo and a remote one. This gives us a daily full backup, with history available, both locally and off-site. Thanks to bup our 4T of data gets deduplicated to about 270G, only a couple of megs get transferred every night, and thanks to git-annex everything just works, like it should!
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlOc-EOD5ZyggsAp6lOnU7x5MxizwLtUXA"
 nickname="Hendrik"
 subject="more success"
 date="2013-06-27T14:09:47Z"
 content="""
Hi,

I have moved all my photographs to an annex running on a Ubuntu file server while doing photo editing, presenting etc. on my MacBook. The annex is great in selectively moving files between these machines for editing and presentation purposes. I started using it around christmas season last year and added a glacier \"backup\" recently. My statistics: 6 repositories in total, more than 300GB (dedup'd from 380GB), 60.000 local keys.

What is missing from my point of view (and I would happily join crowd funding): stable windows and android versions, managing selective transfers from the assistant (select folders to be transferred from/to a specific repository), options for running the annex as a server daemon with web GUI (authentication to GUI + fixed URLs, status reports on GUI, etc.). All these would dramatically increase the WAF.
"""]]


code:
Came up with a fix for the gnucash hard linked file problem that makes the
assistant notice the files gnucash writes. This is not full hard link support;
hard linked files still don't cleanly sync around. But new hard links to
files are noticed and added, which is enough to support gnucash.

Spent around 4 hours on reproducing and trying to debug
[[bugs/Hanging_on_install_on_Mountain_lion]]. It seems that recent upgrades
of the OSX build machine led to this problem. And indeed, building with an
older version of Yesod and Warp seems to have worked around the problem. So
I updated the OSX build for the last release. I will have to re-install the
new Yesod on my laptop and investigate further -- is this an OSX specific
problem, or does it affect Linux? Urgh, this is the second hang I've
encountered involving Warp..

Got several nice [[success stories|day_288__success_stories]], but I don't
think I've seen *yours* yet. ;) Please post!


code:
Spent too many hours last night tracking down a bug that caused the webapp
to hang when it got built with the new yesod 1.2 release. Much of that time
was spent figuring out that yesod 1.2 was causing the problem. It turned out to
be a stupid typo in my yesod compatability layer. `liftH = liftH` in
Haskell is an infinite loop, not the stack overflow you get in most
languages. ;)

Even though it's only been a week since the last release,
that was worth pushing a release out for, which I've just done.
This release is essentially all bug fixes (aside from the automatic
ionice and nicing of the daemon).

This website is now available over https. Perhaps more importantly, all the
links to download git-annex builds are https by default.

The [[success_stories|day_288__success_stories]] list is getting really
nice. Only way it could possibly be nicer is if you added your story! Hint. ;)


code:
I've felt for a while that git-annex needed better support for managing
the contents of past versions of files that are stored in the annex. I know
some people get confused about whether git-annex even supports old versions
of files (it does, but you should use indirect mode; direct mode doesn't
guarantee old versions of files will be preserved).

So today I've worked on adding command-line power for managing past
versions: a new `--all` option.

So, if you want to copy every version of every file in your repository to
an archive, you can run `git annex copy --all --to archive`.
Or if you've got a repository on a drive that's dying, you can run
`git annex copy --all --to newdrive`, and then on the new drive, run `git
annex fsck --all` to check all the data.

In a bare repository, `--all` is default, so you can run `git annex get`
inside a bare repository and it will try to get every version of every file
that it can from the remotes.

The tricky thing about `--all` is that since it's operating on objects and
not files, it can't check `.gitattributes` settings, which are tied to the
file name. I worried for a long time that adding `--all` would make
annex.numcopies settings in those files not be honored, and that this would
be a Bad Thing. The solution turns out to be simple: I just didn't
implement `git annex drop --all`! Dropping is the only action that needs to
check numcopies (move can also reduce the number of copies, but explicitly
bypasses numcopies settings).

I also added an `--unused` option. So if you have a repository that has
been accumulating history, and you'd like to move all file contents not
currently in use to a central server, you can run `git annex unused; git
annex move --unused --to origin`


code:
[[!comment format=mdwn
 username="https://me.yahoo.com/a/2grhJvAC049fJnvALDXek.6MRZMTlg--#eec89"
 nickname="John"
 subject="comment 1"
 date="2013-07-03T22:16:32Z"
 content="""
Excellent stuff!  Another thing I'd like to see is that I find myself doing the following all the time:

    git annex copy --to titan --not --in titan

Meaning: copy everything to titan that it doesn't already have.  This operation is very quick.  But I write just:

    git annex copy --to titan

This operation is extremely slow, since it appears to scan through every file in the repository.  Is there a way to have the latter imply the former?  I wouldn't ever want to copy a file that it already has anyway...

"""]]


code:
[[!comment format=mdwn
 username="sfs"
 ip="65.204.1.102"
 subject="See history of a file"
 date="2013-07-07T00:49:35Z"
 content="""
It's great that you are addressing the management of older versions. I am new to git-annex but couldn't find much information about this topic. Is it also possible to make git-annex show you the history of files (with date, size, ...) and where (which repositories) they are stored? Or to retrieve a specific version from a remote repository?

Thanks for this amazing peace of software!
Stephan
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.254.222"
 subject="comment 3"
 date="2013-07-07T17:20:05Z"
 content="""
@john I have considered making copy trust the location log for the remote (which is what your --not --in titan does), but this does change its behavior in a subtly way, and IIRC there were scenarios where this is not desirable.

@sfs the best way to look at older versions of files is to use `git checkout` to check out an older version of the repository. You can then use `git annex get`, `git annex whereis`, etc like you usually would, on the old version of files.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="comment 4"
 date="2013-07-14T16:03:39Z"
 content="""
@john, @joeyh: I like the fact that normal copy takes some time, but that it _verifies_. Default to save, optimize to quick, imo.
"""]]


code:
Actually spread out over several days..

I think I have finally comprehensively dealt with all the wacky system
misconfigurations that can make `git commit` complain and refuse to commit.
The last of these is [a system with a FQDN that doesn't have a dot in it](http://git-annex.branchable.com/bugs/view_logs_fails:_Internal_Server_Error__internal_liftAnnex/).
I personally think git should just use the hostname as-is in the email
address for commits here -- it's better to be robust. Indeed, I think it
would make more sense if `git commit` never failed, unless it ran out of
disk or the repo is corrupt. But anyway, `git-annex
init` will now detect when the commit fails because of this and put a
workaround in place.

Fixed a bug in `git annex addurl --pathdepth` when the url's path was
shorter than the amount requested to remove from it.

Tracked down a bug that prevents git-annex from working on a system with an
old linux kernel. Probably the root cause is that the kernel was built
without EVENTFD support. Found a workaround to get a usable git-annex on
such a system is to build it without the webapp, since that disables the
threaded runtime which triggered the problem.

Dealt with a lot of Windows bugs. Very happy that it's working well enough
that some users are reporting bugs on it in Windows, and with enough detail
that I have not needed to boot Windows to fix them so far. ;)


code:
[[!comment format=mdwn
 username="http://cweiske.de/"
 nickname="cweiske"
 subject="comment 1"
 date="2013-07-09T06:52:11Z"
 content="""
Thanks for the quick bugfix!
"""]]


code:
[[!comment format=mdwn
 username="http://nullroute.eu.org/~grawity/"
 nickname="Mantas"
 subject="comment 2"
 date="2013-07-09T13:47:20Z"
 content="""
git **used to** use the hostname as-is, but that didn't work out well – many people just kept on committing as `joeyh@kremvax.(none)`, which doesn't make for a good email address or a globally-unique identifier (sometimes by being lazy; sometimes by forgetting to carry their Git config over to a new machine).

The stricter check was implemented in [8c5b1ae1](https://github.com/gitster/git/commit/8c5b1ae1b26a7512eb29e75391b8b24c0d0439e7) for git v1.8.3. Setting `$EMAIL` or running `git -c user.email=foo@bar` should override it.
"""]]


code:
[[!comment format=mdwn
 username="http://nullroute.eu.org/~grawity/"
 nickname="Mantas"
 subject="comment 3"
 date="2013-07-09T13:48:49Z"
 content="""
(I was wrong about the version; it appears to be v1.7.11.)
"""]]


code:
Two gpg fixes today. The OSX Mtn Lion builds were pulling in a build of gpg
that wanted a gpg-agent to be installed in /usr/local or it wouldn't work.
I had to build my own gpg on OSX to work around this. I am pondering making
the OSX dmg builds pull down the gpg source and build their own binary,
so issues on the build system can't affect it. But would really rather not,
since maintaining your own version of every dependency on every
OS is hard (pity about there still being so many OS's without sane package
management).

On Android, which I have not needed to touch for a month, gpg was built
with --enable-minimal, which turned out to not be necessary and was
limiting the encryption algorythms included, and led to
interoperability problems for some. Fixed that gpg build too.

Also fixed an ugly bug in the webapp when setting up a rsync repository.
It would configure `~/.ssh/authorized_keys` on the server to force
git-annex-shell to be run. Which doesn't work for rsync. I didn't notice
this before because it doesn't affect ssh servers that already have a ssh
setup that allows accessing them w/o a password.

Spent a while working on a bug that can occur in a non-utf8 locale
when using special characters in the directory name of a ssh remote.
I was able to reproduce it, but have not worked out how to fix it; encoding
issues like this are always tricky.

Added something to the walkthrough to help convince people that yes, you
can use tags and branches with git-annex just like with regular git. One of
those things that is so obvious to the developer writing the docs
that it's hard to realize it will be a point of concern.

Seems like there is a release worth of changes already, so I plan to push
it out tomorrow.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnXybLxkPMYpP3yw4b_I6IdC3cKTD-xEdU"
 nickname="Matt"
 subject="comment 1"
 date="2013-07-09T07:18:09Z"
 content="""
Hi, glad you added the git history and branch info to the walk through.  With a 40,000 foot view some of the wording in the docs (whilst correct) may lead to the wrong impression.  For example, in the unused data section, there is a sense that it seems bad that data can accumulate and doing a \"git rm\" rather than doing a \"git annex drop\" is a naughty thing to do (I know you don't actually say this).

I think it needs to made clear that \"removing\" files is more subtle than in a traditional file-system: that there is a choice of deleting from the working tree so that contents remain and can be recovered or removing the file and contents labeled together (a.k.a \"dropping\").  With dropping so pervasive in the docs its easy to think this \"right\" (and only) way.

I wonder if a \"git annex rm\" which is simply an alias for \"git rm\" might be a good idea for a UI perspective...
"""]]


code:
[[!comment format=mdwn
 username="LP"
 ip="72.227.131.46"
 subject="Just to say.."
 date="2013-07-10T23:01:56Z"
 content="""
These updates are fantastic. They're great demonstrations of transparency in work, fun to read as a kstarter contributor, and interesting as a copypasta coder. Bravo.
"""]]


code:
Got the release out, after fixing test suite and windows build breakage.
This release has all the features on the command line side (--all,
--unused, etc), but several bugfixes on the assistant side, and a lot
of Windows bug fixes.

I've spent this evening adding icons to git-annex on Linux.
Even got the Linux standalone tarball to automatically install icons.


code:
Been keeping several non-coding balls in the air recently, two of which
landed today.

First, Rsync.net is [offering a discount to all git-annex users](http://www.rsync.net/products/git-annex-pricing.html),
at one third their normal price.
"People using git-annex are clueful and won't be a big support burden for us,
so it's a win-win."
The web app will be updated offer the discount when setting up a rsync.net
repository.

Secondly, I've recorded an interview today for the Git Minutes podcast,
about git-annex. Went well, looking forward to it going up, probably on
Monday.


code:
Surprise! I'm running a new crowdfunding campaign, which I hope will fund
several more months of git-annex development.

<https://campaign.joeyh.name/>

Please don't feel you have to give, but if you do decide to, give
generously. ;) I'm accepting both Paypal and Bitcoin (via CoinBase.com),
and have some rewards that you might enjoy.

----

I came up with two lists of things I hope this campaign will fund.
These are by no means complete lists. First, some general features and
development things:

* Integrate better with Android.
* Get the assistant and webapp ported to Windows.
* Refine the automated stress testing tools to find and fix more problems
  before users ever see them.
* Automatic recovery. Cosmic ray flipped a bit in a file?
  USB drive corrupted itself? The assistant should notice these problems,
  and fix them.
* Encourage more contributions from others. For example, improve the
  special remote plugin interface so it can do everything the native Haskell
  interface can do. Eight new cloud storage services were added this year
  as plugins, but we can do better!
* Use deltas to reduce bandwidth needed to transfer modified versions of files.

Secondly, some things to improve security:

* Add easy support for encrypted git repositories
  using [git-remote-gcrypt](https://github.com/blake2-ppc/git-remote-gcrypt),
  so you can safely push to a repository on a server you don't control.
* Add support for setting up and using GPG keys in the webapp.
* Add protection to the XMPP protocol to guard against man in the middle
  attacks if the XMPP server is compromised. Ie, Google should not be able to
  learn about your git-annex repository even if you're using their servers.
* To avoid leaking even the size of your encrypted files to
  cloud storage providers, add a mode that stores fixed size chunks.

It will also, of course, fund ongoing bugfixing, support, etc.


code:
[[!comment format=mdwn
 username="EskildHustvedt"
 ip="80.202.103.73"
 subject="Shipping costs"
 date="2013-07-15T05:07:07Z"
 content="""
Hi, it doesn't mention anything about offsetting shipping costs for people located outside of the US (like I am). Any chance you could add that?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.152.246.187"
 subject="comment 2"
 date="2013-07-15T05:26:54Z"
 content="""
Shipping is minimal, either $1 or $1.50 outside the US. It's factored into the reward levels, basically.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawm3RqHNOKdnMCRF5V4306ON55XyJtQNlpY"
 nickname="Marco"
 subject="BitSync"
 date="2013-07-15T13:11:03Z"
 content="""
How about using torrent to retrive files from a annex, something like BitSync would be great. Nice job Joey! Thank you.
"""]]


code:
[[!comment format=mdwn
 username="https://dzsino.myopenid.com/"
 nickname="dzsino"
 subject="mo' stickers, less problems :)"
 date="2013-07-16T09:56:03Z"
 content="""
Hi Joey, since shipping would burn 10-15% of my funding, I added another $10, so you can just batch the stickers together. I hope it's cheaper this way.. Thanks!
"""]]


code:
It looks like I'm funded for at least the next 9 months! It would still be
nice to get to a year. ;) <https://campaign.joeyh.name/>

Working to get caught up on recent bug reports..

Made `git annex uninit` not nuke anything that's left over in
`.git/annex/objects` after unannexing all the files. After all, that could
be important old versions of files or deleted file, and just because the
user wants to stop using git-annex, doesn't mean git-annex shouldn't try to
protect that data with its dying breath. So it prints out some suggestions
in this case, and leaves it up to the user to decide what to do with the
data.

Fixed the Android autobuilder, which had stopped including the webapp.

Looks like another autobuilder will be needed for OSX 10.9.


code:
[[!comment format=mdwn
 username="http://grossmeier.net/"
 nickname="greg"
 subject="uninit"
 date="2013-07-16T23:43:38Z"
 content="""
Hah, that bit me in the butt when I first started experimenting with git-annex. I didn't lose anything, but it taught me a very important lesson (that I've had to learn many a time).
"""]]


code:
Theme today seems to be fun with exceptions.

Fixed an uncaught exception that could crash the assistant's Watcher thread
if just the right race occurred.

Also fixed it to not throw an exception if another process is
already transferring a file. What this means is that if you run multiple
`git annex get` processes on the same files, they'll cooperate in each
picking their own files to get and download in parallel. (Also works for
copy, etc.) Especially useful when downloading from an encrypted remote,
since often one process will be decrypting a file while the other is
downloading the next file. There is still room for improvement here;
a -jN option could better handle ensuring N downloads ran concurrently, and
decouple decryption from downloading. But it would need the output layer to
be redone to avoid scrambled output. (All the other stuff to make parallel
git-annex transfers etc work was already in place for a long time.)

----

Campaign update: Now funded for nearly 10 months, and aiming for a year.
<https://campaign.joeyh.name/>


code:
Succeeded fixing a few bugs today, and followed up on a lot of other ones..

Fixed checking when content is present in a non-bare repository accessed via
http.

My changes a few days ago turned out to make uninit leave hard links behind
in .git/annex. Luckily the test suite caught this bug, and it was easily
fixed by making uninit delete objects with 2 or more hard links at the end.


code:
Last night I got `git annex watch` to also handle deletion of files.
This was not as tricky as feared; the key is using `git rm --ignore-unmatch`,
which avoids most problematic situations (such as a just deleted file
being added back before git is run).

Also fixed some races when `git annex watch` is doing its startup scan of
the tree, which might be changed as it's being traversed. Now only one
thread performs actions at a time, so inotify events are queued up during
the scan, and dealt with once it completes. It's worth noting that inotify
can only buffer so many events .. Which might have been a problem except
for a very nice feature of Haskell's inotify interface: It has a thread
that drains the limited inotify buffer and does its own buffering.

----

Right now, `git annex watch` is not as fast as it could be when doing
something like adding a lot of files, or deleting a lot of files.
For each file, it currently runs a git command that updates the index.
I did some work toward coalescing these into one command (which `git annex`
already does normally). It's not quite ready to be turned on yet,
because of some races involving `git add` that become much worse
if it's delayed by event coalescing.

----

And races were the theme of today. Spent most of the day really
getting to grips with all the fun races that can occur between
modification happening to files, and `git annex watch`. The [[inotify]]
page now has a long list of known races, some benign, and several,
all involving adding files, that are quite nasty.

I fixed one of those races this evening. The rest will probably involve
moving away from using `git add`, which necessarily examines the file
on disk, to directly shoving the symlink into git's index.

BTW, it turns out that `dvcs-autosync` has grappled with some of these same
races: <http://comments.gmane.org/gmane.comp.version-control.home-dir/665>
I hope that `git annex watch` will be in a better place to deal with them,
since it's only dealing with git, and with a restricted portion of it
relevant to git-annex.

It's important that `git annex watch` be rock solid. It's the foundation
of the git annex assistant. Users should not need to worry about races
when using it. Most users won't know what race conditions are. If only I
could be so lucky!


code:
git-annex has a new nicer versions of its [[logo]], thanks to John Lawrence.

Finally tracked down a week-old bug about the watcher crashing. It turned
out to crash when it encountered a directory containing a character that's
invalid in the current locale. I've noticed that 'ü' is often the character I
get bug reports about. After reproducing the bug I quickly tracked it down
to code in the haskell hinotify library, and sent in a patch.

Also uploaded a fixed hinotify to Debian, and deployed it to all 3 of the
autobuilder chroots. That took much more time than actually fixing the bug.
Quite a lot of yak shaving went on actually. Oh well. The Linux
autobuilders are updated to use Debian unstable again, which is nice.

Fixed a bug that prevented annex.diskreserve to be honored when storing
files encrypted in a directory special remote.

Taught the webapp the difference between initializing a new special remote
and enabling an existing special remote, which fixed some bad behavior when
it got confused.

----

And then for the really fun bug of the day! A user sent me a large file
which badly breaks git annex add. Adding the file causes a symlink to be
set up, but the file's content is not stored in the annex. Indeed, it's
deleted. This is the first data loss bug since January 2012.

Turns out it was caused by the code that handles the dummy files git uses
in place of symlinks on FAT etc filesystems. Code that had no business
running when `core.symlinks=true`. Code that was prone to false positives
when looking at a tarball of a git-annex repository. So I put in multiple
fixes for this bug. I'll be making a release on Monday.

----

Today's work was sponsored by Mikhail Barabanov. Thanks, Mikhail!


code:
[[!comment format=mdwn
 username="https://launchpad.net/~mikapflueger"
 nickname="mikapflueger"
 subject="The dreaded "ü""
 date="2013-07-21T20:56:30Z"
 content="""
Yeah, with the last name of \"Pflüger\" I also usually find bugs having to do with the letter \"ü\". (-;
I guess it is simply one of the more frequent german umlauts, especially in names (the classic \"Müller\", which is one of the most common last names in Germany). And apparently the German cabal strikes here again.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmNu4V5fvpLlBhaCUfXXOB0MI5NXwh8SkU"
 nickname="Adam"
 subject="Horns?"
 date="2013-07-31T19:53:51Z"
 content="""
I don't necessarily mean this as a criticism, just an observation. I wasn't a big fan of the old logo, but this new one looks like it has horns. Or a bit like a pitchfork.
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.0.21"
 subject="comment 3"
 date="2013-07-31T20:03:13Z"
 content="""
I think the arrows are a bit too pointy. If someone wants to adjust the svg to make the arrows have a bit more of a rounded feel like in the old logo, that'd be fine by me.
"""]]


code:
No release today after all. Unexpected bandwidth failure. Maybe in a few
days..

Got unannex and uninit working in direct mode. This is one of the more
subtle parts of git-annex, and took some doing to get it right.
Surprisingly, unannex in direct mode actually turns out to be faster than
in indirect mode. In direct mode it doesn't have to immediately commit the
unannexing, it can just stage it to be committed later.

Also worked on the ssh connection caching code. The perrennial problem with
that code is that the fifo used to communicate with ssh has a small limit
on its path, somewhere around 100 characters. This had caused problems when
the hostname was rather long. I found a way to avoid needing to be able to
reverse back from the fifo name to the hostname, and this let me take the
md5sum of long hostnames, and use that shorter string for the fifo.

Also various other bug followups.

----

[Campaign](https://campaign.joeyh.name/) is almost to 1 year!


code:
Got the release out.

I've been working on fleshing out
the [[timeline|/design/assistant]] for the next year.
Including a fairly detailed set of things I want to do around
[[/design/assistant/disaster_recovery]] in the assistant.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmnG4EuvZWse5hvgrl0XAK-U61e-0iGaao"
 nickname="David"
 subject="Broken dependencies on Debian wheezy "
 date="2013-07-24T07:21:18Z"
 content="""
Hi Joey,
Having installed this release (x86_64) on Debian wheezy I get the following error when I try to run git-annex:

    git-annex: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found (required by git-annex)
    git-annex: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found (required by $HOME/.local/share/git-annex//usr/lib/x86_64-linux-gnu/libidn.so.11)

etc. for at least a dozen libraries. Different libs try to find different glibc version (2.14, 2.15, 2.16, 2.17). Previous releases were linked with GLIBC_2.13.

"""]]


code:
Seems I forgot why I was using debian stable chroots to make the
autobuilds: Lots of people still using old glibc version. Had to rebuild
the stable chroots that I had upgraded to unstable. Wasted several hours..
I was able to catch up on recent traffic in between.

Was able to reproduce a bug where `git annex initremote` hung with some
encrypted special remotes. Turns out to be a deadlock when it's not built
with the threaded GHC runtime. So I've forced that runtime to be used.


code:
The big news: Important behavior change in `git annex dropunused`. Now it
checks, just like `git annex drop`, that it's not dropping the last copy of
the file. So to lose data, you have to use `--force`. This continues the
recent theme of making git-annex hold on more tenaciously to old data, and
AFAIK it was the last place data could be removed without `--force`.

Also a nice little fix to `git annex unused` so it doesn't identify
temporary files as unused if they're being used to download a file.
Fixing it was easy thanks to all the transfer logs and locking
infrastucture built for the assistant.

Fixed a bug in the assistant where even though syncing to a network
remote was disabled, it would still sync with it every hour, or whenever
a network connection was detected.

Working on some direct mode scalability problems when thousands of the
identical files are added. Fixing this may involvie replacing the current
simple map files with something more scalable like a sqllite database.

While tracking that down, I also found a bug with adding a ton of files
in indirect mode, that could make the assistant stall.
Turned out to be a laziness problem. (Worst kind of Haskell bug.) Fixed.

----

Today's sponsor is my sister, Anna Hess, who incidentially just put
the manuscript of her latest ebook in the family's annex prior to its
publication on Amazon this weekend.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlUbH3eytydcwlWqv8oauE2Jg4NwcV9uA0"
 nickname="Anna"
 subject="A good day to be the sponsor!"
 date="2013-07-27T12:31:27Z"
 content="""
Thanks for mentioning me!  I like being responsible for not allowing things to be deleted accidentally --- just my kind of feature. :-)

As a shameless plug, here's the ebook mentioned above for those of you not privy to the family annex: http://www.amazon.com/Root-Cellar-Vegetables-Simplicity-ebook/dp/B00E6EGS0M/
"""]]


code:
Worked on 3 interesting bugs today. One I noticed myself while doing tests
with adding many thousands of files yesterday. Assistant was delaying
making a last commit of the batch of files, and would only wake up and
commit them after a further change was made. Turns out this bug was
introduced in April while improving commit batching when making very large
commits. I seem to remember someone mentioning this problem at some point,
but I have not been able to find a bug report to close.

Also tried to reproduce [[this_bug|bugs/utf8]]. Frustrating, because I'm
quite sure I have made changes that will avoid it happening again,
but since I still don't know what the root cause was, I can't let it go.

The last bug is
[[non-repos_in_repositories_list___40__+_other_weird_output__41___from_git_annex_status]]
and is a most strange thing. Still trying to get a handle on multiple
aspects of it.

Also various other bug triage. Down to only 10 messages in my git-annex
folder.  That included merging about a dozen bugs about
local pairing, that all seem to involve git-annex-shell not being found in
path. Something is up with that..


code:
Technically offtopic, but did a fun side project today:
<http://joeyh.name/blog/entry/git-annex_as_a_podcatcher/>


code:
Back to bug squashing. Fixed several, including a long-standing problem on
OSX that made the app icon seem to "bounce" or not work. Followed up on a
bunch more.

The 4.20130723 git-annex release turns out to have broken support for
running on crippled filesystems (Android, Windows). `git annex sync` will
add dummy symlinks to the annex as if they were regular files, which is
not good!
[Recovery instructions](http://git-annex.branchable.com/bugs/regression_in_direct_mode_on_windows_:_weird___96__git_annex_sync__96___behavior/#comment-5d80649f9da85ac2fb505445a41207f5)
I've updated the Android and Windows builds and recommend an immediate upgrade.
Will make a formal release on Friday.

Spent some time improving the test suite on Windows, to catch this bug,
and fix a bug that was preventing it from testing `git annex sync` on
Windows.

----

I am getting very frustrated with this "unknown UUID" problem that a dozen
people have reported. So far nobody has given me enough information to
reproduce the problem. It seems to have something to do with
`git-annex-shell` not being found on the remote system that has been either
local paired with or is being used as a ssh server, but I don't yet
understand what. I have spent hours today trying various scenarios to break
git-annex and get this problem to happen.

I certainly can improve the webapp's behavior when a repository's UUID is
not known. The easiest fix would be to simply not display such
repositories. Or there could be a UI to try to get the UUID.
But I'm more interested in fixing the core problem than putting
in a UI bandaid.


code:
Turns out ssh-agent is the cause of the unknown UUID bug! I got a tip
about this from a user, and was quickly able to reproduce the bug that had
eluded me so long. Anyone who has run `ssh-add` and is using ssh-agent
would see the bug.

It was easy enough to fix as it turns out. Just need to set IdentitiesOnly
in .ssh/config where git-annex has set up its own IdentityFile to ensure
that its special purpose ssh key is used rather than whatever key the
ssh-agent has loaded into it. I do wonder why ssh behaves this way -- why
would I set an IdentityFile for a host if I didn't want ssh to use it?

Spent the rest of the day cleaning up after the bug. Since this affects so
many people, I automated the clean up process. The webapp will
detect repositories with this problem, and the user just has to click to
clean up. It'll then correct their .ssh/config and re-enable the
repository.


code:
[[!comment format=mdwn
 username="https://launchpad.net/~mikapflueger"
 nickname="mikapflueger"
 subject="Thank you so much!"
 date="2013-08-01T01:13:51Z"
 content="""
This behaviour of ssh has bugged me for years now (I even think it is an information leak – ssh offers basically all my public ssh keys to every server I try to connect to, even though that public key might be confidential). Although it had nothing to do with git-annex, you helped me a _lot_ by pointing me towards IdentitiesOnly. Really a nice setting!

Cheers,

Mika
"""]]


code:
Today was a nice reminder that there are no end of bugs lurking in filename
handling code.

First, fixed a bug that prevented git-annex from adding
filenames starting with ":", because that is a special character to git.

Second, discovered that git 1.8.4 rc0 has changed `git-cat-file --batch` in
a way that makes it impossible to operate on filenames containing spaces.
This is, IMHO, a reversion, so hopefully my
[bug report](http://bugs.debian.org/718517) will get it fixed.

Put in a workaround for that, although using the broken version of git
with a direct mode repository with lots of spaces in file or directory
names is going to really slow down git-annex, since it often has to fork a
new git cat-file process for each file.

Release day tomorrow..


code:
Got the release out, with rather a lot of fiddling to fix broken builds on
various platforms.

Also released a backport to Debian stable. This backport has the assistant,
although without WebDAV support. Unfortunately it's an old version from
May, since ghc transitions and issues have kept newer versions out of
testing so far. Hope that will clear up soon (probably by dropping haskell
support for s390x), and I can update it to a newer version.
If nothing else it allows using direct mode with Debian stable.

Pleased that the git-cat-files bug was quickly fixed by Peff and has
already been pulled into Junio's release tree!

----

This evening, I've added an interface around the new improved
`git check-ignore` in git 1.8.4. The assistant can finally honor `.gitignore`
files!


code:
[[!comment format=mdwn
 username="http://adamspiers.myopenid.com/"
 nickname="Adam"
 subject="This makes me happy :-)"
 date="2013-08-04T00:46:25Z"
 content="""
Glad my work was worth the effort. Thanks for that.
"""]]


code:
Made two big improvements to the Windows port, in just a few hours.
First, got gpg working, and encrypted special remotes work on Windows.
Next, fixed a permissions problem that was breaking removing files
from directory special remotes on Windows.
(Also cleaned up a lot of compiler warnings on Windows.)

I think I'm almost ready to move the Windows port from alpha to beta
status. The only really bad problem that I know of with using it is that
due to a lack of locking, it's not safe to run multiple git-annex
commands at the same time in Windows.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawlpSOjMH7Iaz56v6Pr9KCFSpbvMXvg-y9o"
 nickname="Dominik"
 subject="That's amazing..."
 date="2013-08-17T11:18:05Z"
 content="""
...so pretty soon my bi-directional Mac <-> Win Git-Annex-Assitant sync via rsync.net will work without a VM on Windows :-)
"""]]


code:
Wow, 11 days off! I was busy with first dentistry and then DebConf.

Yesterday I [visited CERN](http://joeyh.name/blog/entry/words_fail_me/) and
got to talk with some of their IT guys about how they manage their tens of
petabytes of data. Interested to hear they also have the equivilant of a
per-subdirectory annex.numcopies setting. OTOH, they have half a billion
more files than git's index file is likely to be able to scale to support. ;)

Pushed a release out today despite not having many queued changes.
Also, I got git-annex migrated to Debian testing, and so was also
able to update the wheezy backport to a just 2 week old version.

Today is also the last day of the [campaign](https://campaign.joeyh.name/)!

----

There has been a ton of discussion about git-annex here at DebConf,
including 3 BoF sessions that mostly focused on it, among other git stuff.
Also, RichiH will be presenting his
"[Gitify Your Life](http://penta.debconf.org/dc13_schedule/events/1025.en.html)"
talk on Friday; you can catch it on the [live stream](http://blog.debconf.org/blog/2013/08/14#hl_dc13_recordings).

I've also had a continual stream of in-person bug and feature requests.
(Mostly features.)
These have been added to the wiki and I look forward to working on that
backlog when I get home.

As for coding, I am doing little here, but I do have a branch cooking that
adds some options to `git annex import` to control handling of duplicate
files.


code:
Back home. I have some 170 messages of backlog to attend to. Rather than
digging into that on my first day back, I spent some time implementing some
new features.

`git annex import` has grown three options that help managing importing of
duplicate files in different ways. I started work on that last week, but
didn't have time to find a way to avoid the `--deduplicate` option
checksumming each imported file twice. Unfortunately, I have still not
found a way I'm happy with, so it works but is not as efficient as it could
be.

`git annex mirror` is a new command suggested to me by someone at DebConf
(they don't seem to have filed the requested todo). It arranges for two
repositories to contain the same set of files, as much as possible (when
numcopies allows). So for example, `git annex mirror --to otherdrive`
will make the otherdrive remote have the same files present and not present
as the local repository.

I am thinking about expanding `git annex sync` with an option to also sync
data. I know some find it confusing that it only syncs the git metadata
and not the file contents. That still seems to me to be the best and most
flexible behavior, and not one I want to change in any case since
it would be most unexpected if `git annex sync` downloaded a lot of stuff
you don't want. But I can see making `git annex sync --data` download
all the file contents it can, as well as uploading all available file
contents to each remote it syncs with. And `git annex sync --data --auto`
limit that to only the preferred content. Although perhaps
these command lines are too long to be usable?

----

With the campaign more or less over, I only have a little over a week
before it's time to dive into the first big item on the roadmap. Hope
to be through the backlog by then.


code:
[[!comment format=mdwn
 username="GLITTAH"
 ip="77.247.181.162"
 subject="comment 1"
 date="2013-08-20T22:42:46Z"
 content="""
I wouldn't worry about the commands being too long.  If they're used often enough, they'll get aliased.

alias GASDA='git annex sync --data --auto'
"""]]


code:
Made some good progress on the backlog today. Fixed some bugs, applied some
patches. Noticing that without me around, things still get followed up
on, to a point, for example incomplete test cases for bugs get corrected so
they work. This is a very good thing. Community!

I had to stop going through the backlog when I got to one message from
Anarcat mentioning [quvi](http://quvi.sourceforge.net/). That turns
out to be just what is needed to implement the often-requested feature
of `git-annex addurl` supporting YouTube and other similar sites. So I
spent the rest of the day making that work. For example:

<pre>
% git annex addurl --fast 'http://www.youtube.com/watch?v=1mxPFHBCfuU&list=PL4F80C7D2DC8D9B6C&index=1'
addurl Star_Wars_X_Wing__Seth_Green__Clare_Grant__and_Mike_Lamond_Join_Wil_on_TableTop_SE2E09.webm ok
</pre>

Yes, that got the video title and used it as the filename, and yes,
I can commit this file and run `git annex get` later, and it will be
able to go download the video! I can even use `git annex fsck --fast`
to make sure YouTube still has my videos. Awesome.

The great thing about quvi is it takes the url to a video webpage, and
returns an url that can be used to download the actual video file. So it
simplifies ugly flash videos as far out of existence as is possible.
However, since the direct url to the video file may not keep working for long.
addurl actually records the page's url, with an added indication that quvi
should be used to get it.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmNu4V5fvpLlBhaCUfXXOB0MI5NXwh8SkU"
 nickname="Adam"
 subject="New YouTube problems"
 date="2013-09-05T03:45:16Z"
 content="""
Joey, you may notice eventually that YouTube is moving some videos, especially long ones, to a new format called DASH, which splits them up into smaller parts; unfortunately, it also delivers the audio as a separate file, requiring something like ffmpeg to merge it with the video.  I don't know anything about quvi, but youtube-dl recently added support for DASH.  It's a Python program that's packaged in Debian, but it can also update itself if you install it locally, which is helpful if Debian is frozen...  Anyway, if you need it in the future, youtube-dl might be helpful.
"""]]


code:
After a couple days plowing through it, my backlog is down to 30 messages
from 150. And most of what's left is legitimate bugs and todo items.

Spent a while today on an ugly file descriptor leak in the assistant's
local pairing listener. This was an upstream bug in the network-multicast
library, so while I've written a patch to fix it, the fix isn't quite
deployed yet. The file descriptor leak happens when the assistant is
running and there is no network interface that supports multicast.
I was able to reproduce it by just disconnecting from wifi.

Meanwhile, guilhem has been working on patches that promise to massively
speed up `git annex unused`! I will be reviewing them tonight.


code:
Today was a day off, really. However, I have a job running to try to
build get a version of ghc-android that works on newer Android releases.

Also, guilhem's `git annex unused` speedup patch landed. The results are
extrordinary -- speedups on the order of 50 to 100 times faster should
not be uncommon. Best of all (for me), it still runs in constant memory!


code:
Spent a while tracking down a bug that causes a crash on OSX when setting
up an XMPP account. I managed to find a small test case that reliably
crashes, and sent it off to the author of the haskell-gnutls bindings,
which had one similar segfault bug fixed before with a similar test case.
Fingers crossed..

Just finished tracking down a bug in the Android app that caused its
terminal to spin and consume most CPU (and presumably a lot of battery).
I introduced this bug when adding the code to open urls written to a fifo,
due to misunderstanding how java objects are created, basically. This bug
is bad enough to do a semi-immediate release for; luckily it's just about
time for a release anyway with other improvements, so in the next few
days..

Have not managed to get a recent ghc-android to build so far.

Guilhem fixed some bugs in `git annex unused`.


code:
Managed to find a minimal, 20 line test case for at least one of the ways
git-annex was hanging with GHC's threaded runtime. Sent it off to
haskell-cafe for analysis.
[thread](http://thread.gmane.org/gmane.comp.lang.haskell.cafe/99334)

Further managed to narrow the bug down to MissingH's use of logging code,
that git-annex doesn't use. [bug report](http://bugs.debian.org/681621).
So, I can at least get around this problem with a modified version of
MissingH. Hopefully that was the only thing causing the hangs I was seeing!


code:
Back home and laptop is fixed.. back to work.

Warmup exercises:

* Went in to make it queue transfers when a broken symlink is received,
  only to find I'd already written code to do that, and forgotten about it.
  Heh. Did check that the git-annex branch is always sent first,
  which will ensure that code always knows where to transfer a key from.
  I had probably not considered this wrinkle when first writing the code;
  it worked by accident.

* Made the assistant check that a remote is known to have a key before
  queueing a download from it.

* Fixed a bad interaction between the `git annex map` command and the
  assistant.

----

Tried using a modified version of `MissingH` that doesn't use `HSLogger`
to make git-annex work with the threaded GHC runtime. Unfortunatly,
I am still seeing hangs in at least 3 separate code paths when
running the test suite. I may have managed to fix one of the hangs,
but have not grokked what's causing the others.

----

I now have access to a Mac OSX system, thanks to Kevin M. I've fixed
some portability problems in git-annex with it before, but today I tested
the assistant on it:

* Found a problem with the kqueue code that prevents incoming pushes from
  being noticed.

  The problem was that the newly added git ref file does not trigger an add
  event. The kqueue code saw a generic change event for the refs directory,
  but since the old file was being deleted and replaced by the new file,
  the kqueue code, which already had the old file in its cache, did not notice
  the file had been replaced.

  I fixed that by making the kqueue code also track the inode of each file.
  Currently that adds the overhead of a stat of each file, which could be
  avoided if haskell exposed the inode returned by `readdir`. Room to
  optimise this later...

* Also noticed that the kqueue code was not separating out file deletions
  from directory deletions. IIRC Jimmy had once mentioned a problem with file
  deletions not being noticed by the assistant, and this could be responsible
  for that, although the directory deletion code seems to handle them ok
  normally. It was making the transfer watching thread not notice when
  any transfers finished, for sure. I fixed this oversight, looking in the
  cache to see if there used to be a file or a directory, and running the
  appropriate hook.

Even with these fixes, the assistant does not yet reliably transfer file
contents on OSX. I think the problem is that with kqueue we're not
guaranteed to get an add event, and a deletion event for a transfer
info file -- if it's created and quickly deleted, the code that
synthensizes those events doesn't run in time to know it existed.
Since the transfer code relies on deletion events to tell when transfers
are complete, it stops sending files after the first transfer, if the
transfer ran so quickly it doesn't get the expected events.

So, will need to work on OSX support some more...


code:
Beating my head against the threaded runtime some more. I can reproduce
one of the hangs consistently by running 1000 git annex add commands
in a loop. It hangs around 1% of the time, reading from `git cat-file`.

Interestingly, `git cat-file` is not yet running at this point --
git-annex has forked a child process, but the child has not yet exec'd it.
Stracing the child git-annex, I see it stuck in a futex. Adding tracing,
I see the child never manages to run any code at all.

This really looks like the problem is once again in MissingH, which uses
`forkProcess`. Which happens to come with a big warning about being very
unsafe, in very subtle ways. Looking at the C code that the newer `process`
library uses when sparning a pipe to a process, it messes around with lots of
things; blocking signals, stopping a timer, etc. Hundreds of lines of C
code to safely start a child process, all doing things that MissingH omits.

That's the second time I've seemingly isolated a hang in the GHC threaded
runtime to MissingH.

And so I've started converting git-annex to use the new `process` library,
for running all its external commands. John Goerzen had mentioned `process`
to me once before when I found a nasty bug in MissingH, as the cool new
thing that would probably eliminate the `System.Cmd.Utils` part of MissingH,
but I'd not otherwise heard much about it. (It also seems to have the
benefit of supporting Windows.)

This is a big change and it's early days, but each time I see a hang, I'm
converting the code to use `process`, and so far the hangs have just gone
away when I do that.

---

Hours later... I've converted *all* of git-annex to use `process`.

In the er, process, the `--debug` switch stopped printing all the commands
it runs. I may try to restore that later.

I've not tested everything, but the test suite passes, even when
using the threaded runtime. **MILESTONE**

Looking forward to getting out of these weeds and back to useful work..

---

Hours later yet.... The `assistant` branch in git now uses the threaded
runtime. It works beautifully, using proper threads to run file transfers
in.

That should fix the problem I was seeing on OSX yesterday. Too tired to
test it now.

--

Amazingly, all the assistant's own dozen or so threads and thread
synch variables etc all work great under the threaded runtime. I had
assumed I'd see yet more concurrency problems there when switching to it,
but it all looks good. (Or whatever problems there are are subtle ones?)

I'm very relieved. The threaded logjam is broken! I had been getting
increasingly worried that not having the threaded runtime available would
make it very difficult to make the assistant perform really well, and cause
problems with the webapp, perhaps preventing me from using Yesod.

Now it looks like smooth sailing ahead. Still some hard problems, but
it feels like with inotify and kqueue and the threaded runtime all
dealt with, the really hard infrastructure-level problems are behind me.


code:
Today I worked on the race conditions, and fixed two of them. Both
were fixed by avoiding using `git add`, which looks at the files currently
on disk. Instead, `git annex watch` injects symlinks directly into git's
index, using `git update-index`.

There is one bad race condition remaining. If multiple processes have a
file open for write, one can close it, and it will be added to the annex.
But then the other can still write to it.

----

Getting away from race conditions for a while, I made `git annex watch`
not annex `.gitignore` and `.gitattributes` files.

And, I made it handle running out of inotify descriptors. By default,
`/proc/sys/fs/inotify/max_user_watches` is 8192, and that's how many
directories inotify can watch. Now when it needs more, it will print
a nice message showing how to increase it with `sysctl`.

FWIW, DropBox also uses inotify and has the same limit. It seems to not
tell the user how to fix it when it goes over. Here's what `git annex
watch` will say:

	Too many directories to watch! (Not watching ./dir4299)
	Increase the limit by running:
	  echo fs.inotify.max_user_watches=81920 | sudo tee -a /etc/sysctl.conf; sudo sysctl -p


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkmtR6oVColYKoU0SjBORLDGrwR10G-mKo"
 nickname="Jo-Herman"
 subject="Dropbox Inotify"
 date="2012-06-06T22:03:29Z"
 content="""
Actually, Dropbox giver you a warning via libnotify inotify. It tends to go away too quickly to properly read though, much less actually copy down the command...
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.252.8.36"
 subject="comment 2"
 date="2012-06-06T23:25:57Z"
 content="""
When I work on the [[webapp]], I'm planning to make it display this warning, and any other similar warning messages that might come up.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnBJ6Dv1glxzzi4qIzGFNa6F-mfHIvv9Ck"
 nickname="Jim"
 subject="Wording"
 date="2012-06-07T03:43:19Z"
 content="""
For the unfamiliar, it's hard to tell if a command like that would persist.  I'd suggest being as clear as possible, e.g.:

    Increase the limit for now by running:
      sudo sysctl fs.inotify.max_user_watches=81920
    Increase the limit now and automatically at every boot by running:
      echo fs.inotify.max_user_watches=81920 | sudo tee -a /etc/sysctl.conf; sudo sysctl -p

"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.252.8.36"
 subject="comment 4"
 date="2012-06-07T04:48:15Z"
 content="""
Good thought Jim. I've done something like that.
"""]]


code:
Really productive day today, now that I'm out of the threaded runtime
tarpit!

First, brought back `--debug` logging, better than before! As part of that, I
wrote some 250 lines of code to provide a IMHO more pleasant interface to
`System.Process` (itself only 650 lines of code) that avoids all the
low-level setup, cleanup, and tuple unpacking. Now I can do things like
write to a pipe to a process, and ensure it exits nonzero, this easily:

	withHandle StdinHandle createProcessSuccess (proc "git" ["hash-object", "--stdin"]) $ \h ->
		hHutStr h objectdata

My interface also makes it easy to run nasty background processes,
reading their output lazily.

	lazystring <- withHandle StdoutHandle createBackgroundProcess (proc "find" ["/"]) hGetContents

Any true Haskellers are shuddering here, I really should be using
conduits or pipes, or something. One day..

----

The assistant needs to detect when removable drives are attached, and
sync with them. This is a reasonable thing to be working on at this point,
because it'll make the currently incomplete data transfer code fully usable
for the sneakernet use case, and firming that up will probably be a good
step toward handing other use cases involving data transfer over the
network, including cases where network remotes are transientely available.

So I've been playing with using dbus to detect mount events.
There's a very nice Haskell library to use dbus.

This simple program will detect removable drives being mounted, and
works on Xfce (as long as you have automounting enabled in its
configuration), and should also work on Gnome, and, probably, KDE:

[[!format haskell """
{-# LANGUAGE OverloadedStrings #-}

import Data.List (sort)
import DBus
import DBus.Client
import Control.Monad

main = do
	client <- connectSession

	listen client mountadded $ \s ->
		putStrLn (show s)

	forever $ getLine -- let listener thread run forever

	where
		mountadded = matchAny
			{ matchInterface = Just "org.gtk.Private.RemoteVolumeMonitor"
			, matchMember = Just "MountAdded"
			}
"""]]

(Yeah... "org.gtk.Private.RemoteVolumeMonitor". There are so
many things wrong with that string. What does gtk have to do with
mounting a drive? Why is it Private? Bleagh. Should I only match
the "MountAdded" member and not the interface? Seems everyone who does
this relies on google to find other people who have cargo-culted it,
or just runs `dbus-monitor` and picks out things.
There seems to be no canonical list of events. Bleagh.)

----

Spent a while shaving a yak of needing a `getmntent` interface in Haskell.
Found one in a hsshellscript library; since that library is not packaged
in Debian, and I don't really want to depend on it, I extracted just
the mtab and fstab parts of it into a little library in git-annex.

----

I've started putting together a MountWatcher thread. On systems without
dbus (do OSX or the BSDs have dbus?), or if dbus is not running, it polls
`/etc/mtab` every 10 seconds for new mounts. When dbus is available,
it doesn't need the polling, and should notice mounts more quickly.

Open question: Should it still poll even when dbus is available? Some of us
like to mount our own drives, by hand and may have automounting disabled. It'd
be good if the assistant supported that. This might need a
`annex.no-dbus` setting, but I'd rather avoid needing such manual
configuration.

One idea is to do polling in addition to dbus, if `/etc/fstab` contains
mount points that seem to be removable drives, on which git remotes lives.
Or it could always do polling in addition to dbus, which is just some extra
work. Or, it could try to introspect dbus to see if mount events will
be generated.

The MountWatcher so far only detects new mounts and prints out what
happened. Next up: Do something in response to them.

This will involve manipulating the Annex state to belatedly add the Remote
on the mount point.. tricky. And then, for Git Remotes, it should pull/push
the Remote to sync git data. Finally, for all remotes, it will need to
queue Transfers of file contents from/to the newly available Remote.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
 nickname="Richard"
 subject="dbus vs polling"
 date="2012-07-19T17:38:40Z"
 content="""
I am running KDE, dbus, and I like to mount drives by hand; all that without bothering to set up /etc/fstab... Often from CLI so I don't even see the notifications for new drives.

Especially while on the road, I will use no KDE, but just a tty or three. Still, dbus will be running.

Long story short, I would need polling for the assistant to work flawlessly in all use cases.
"""]]


code:
[[!comment format=mdwn
 username="hamish"
 ip="203.0.139.24"
 subject="dbus vs polling "
 date="2012-07-22T07:13:37Z"
 content="""
I, too, am running a dbus but like to hand mount my filesystems.  However, I'd imagine that I am both a minority and that my minority could like the extra control, so perhaps even a \"re-read the mtab /now/\" command that can be manually run after something is  manually mounted would suffice

Is it not possible to use inotify on the mtab?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.154.4.169"
 subject="comment 3"
 date="2012-07-22T16:03:52Z"
 content="""
How did I not think about using my favorite hammer on this problem too? But, no, /proc/mounts cannot be watched with inotify it seems, and of course the BSDs don't seem to have a file at all.

I think the dbus stuff is sorted out for manual users, see later blog entries.
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
 nickname="Jimmy"
 subject="comment 2"
 date="2012-07-19T18:43:30Z"
 content="""
Joey, yes dbus is available from macports and homebrew, it's not installed by default (or as a dependancy) for most packages in macports.
"""]]


code:
[[!comment format=mdwn
 username="http://smcv.pseudorandom.co.uk/"
 nickname="smcv"
 subject="comment 5"
 date="2012-07-31T09:58:13Z"
 content="""
It's `org.gtk` because gvfs, GLib and Gtk are all products of gtk.org (there is no separate glib.org).

It's `Private` because the gvfs developers don't consider its D-Bus interface to be stable API, I believe. The official API for it is the C API in GIO, part of GLib.

To poll for changes to /proc/mounts on Linux, you open it for reading and poll() for POLLERR (not sure why it gets to be different, but it is). See gio/gunixmounts.c in recent GLib.
"""]]


code:
I made the MountWatcher only use dbus if it sees a client connected to dbus
that it knows will send mount events, or if it can start up such a client
via dbus. (Fancy!) Otherwise it falls back to polling. This should be enough
to support users who manually mount things -- if they have gvfs
installed, it'll be used to detect their manual mounts, even when a desktop
is not running, and if they don't have gvfs, they get polling.

Also, I got the MountWatcher to work with KDE. Found a dbus event that's
emitted when KDE mounts a drive, and this is also used. If anyone with
some other desktop environment wants me to add support for it, and it uses
dbus, it should be easy: Run `dbus-monitor`, plug in a drive, get
it mounted, and send me a transcript.

Of course, it'd also be nice to support anything similar on OSX that can
provide mount event notifications. Not a priority though, since the polling
code will work.

---

Some OS X fixes today..

* Jimmy pointed out that my `getmntent` code broke the build on OSX again.
  Sorry about that.. I keep thinking Unix portability nightmares are a 80's
  thing, not a 2010's thing. Anyway, adapted a lot of hackish C code
  to emulate `getmntent` on BSD systems, and it seems to work. (I actually
  think the BSD interface to this is saner than Linux's, but I'd rather have
  either one than both, sigh..)
* Kqueue was blocking all the threads on OSX. This is fixed, and the
  assistant seems to be working on OSX again.

----

I put together a preliminary page thanking everyone who contributed to the
git-annex Kickstarter. [[/assistant/thanks]] The wall-o-names is scary crazy humbling.

----

Improved `--debug` mode for the assistant, now every thread says whenever
it's doing anything interesting, and also there are timestamps.

----

Had been meaning to get on with syncing to drives when they're mounted, but
got sidetracked with the above. Maybe tomorrow. I did think through it
in some detail as I was waking up this morning, and think I have a pretty
good handle on it.


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawnYD2ZzaOz-0anQDrN-Hg8Tvh5_C7wtStk"
 nickname="roucaries"
 subject="Portability"
 date="2012-07-21T20:31:32Z"
 content="""
For portability why not using gnulib ? It will ease porting to windows BTW.

Bastien
"""]]


code:
Made the MountWatcher update state for remotes located in a drive that
gets mounted. This was tricky code. First I had to make remotes declare
when they're located in a local directory. Then it has to rescan git
configs of git remotes (because the git repo mounted at a mount point may
change), and update all the state that a newly available remote can affect.

And it works: I plug in a drive containing one of my git remotes, and the
assistant automatically notices it and syncs the git repositories.

---

But, data isn't transferred yet. When a disconnected remote becomes
connected, keys should be transferred in both directions to get back into
sync.

To that end, added Yet Another Thread; the TransferScanner thread
will scan newly available remotes to find keys, and queue low priority
transfers to get them fully in sync.

(Later, this will probably also be used for network remotes that become
available when moving between networks. I think network-manager sends
dbus events it could use..)

This new thread is missing a crucial peice, it doesn't yet have a way to
find the keys that need to be transferred. Doing that efficiently (without
scanning the whole git working copy) is Hard. I'm considering design
possibilities..


code:
Milestone: I can run `git annex assistant`, plug in a USB drive, and it
automatically transfers files to get the USB drive and current repo back in
sync.

I decided to implement the naive scan, to find files needing to be
transferred. So it walks through `git ls-files` and checks each file
in turn. I've deferred less expensive, more sophisticated approaches to later.

I did some work on the TransferQueue, which now keeps track of the length
of the queue, and can block attempts to add Transfers to it if it gets too
long. This was a nice use of STM, which let me implement that without using
any locking.

[[!format haskell """
atomically $ do
        sz <- readTVar (queuesize q)
        if sz <= wantsz
                then enqueue schedule q t (stubInfo f remote)
                else retry -- blocks until queuesize changes
"""]]

Anyway, the point was that, as the scan finds Transfers to do,
it doesn't build up a really long TransferQueue, but instead is blocked
from running further until some of the files get transferred. The resulting
interleaving of the scan thread with transfer threads means that transfers
start fairly quickly upon a USB drive being plugged in, and kind of hides
the innefficiencies of the scanner, which will most of the time be
swamped out by the IO bound large data transfers.

---

At this point, the assistant should do a good job of keeping repositories
in sync, as long as they're all interconnected, or on removable media
like USB drives. There's lots more work to be done to handle use cases
where repositories are not well-connected, but since the assistant's
[[syncing]] now covers at least a couple of use cases, I'm ready to move
on to the next phase. [[Webapp]], here we come!


code:
After an all-nighter, I have `git annex webapp` launching a WebApp!

It doesn't do anything useful yet, just uses Yesod to display a couple of
hyperlinked pages and a favicon, securely.

The binary size grew rather alarmingly, BTW. :) Indeed, it's been growing
for months..

	-rwxr-xr-x 1 root root 9.4M Jul 21 16:59 git-annex-no-assistant-stripped
	-rwxr-xr-x 1 joey joey  12M Jul 25 20:54 git-annex-no-webapp-stripped
	-rwxr-xr-x 1 joey joey  17M Jul 25 20:52 git-annex-with-webapp-stripped

----

Along the way, some Not Invented Here occurred:

I didn't use the yesod scaffolded site, because it's a lot of what
seems mostly to be cruft in this use case. And because I don't like
code generated from templates that people are then expected to edit. Ugh.
That's my least favorite part of Yesod. This added some pain, since
I had to do everything the hard way.

I didn't use [wai-handler-launch](http://hackage.haskell.org/package/wai-handler-launch)
because:

  * It seems broken on IPv6 capable machines (it always opens
    `http://127.0.0.1:port/` even though it apparently doesn't always
    listen there.. I think it was listening on my machine's ipv6 address
    instead. I know, I know; I should file a bug about this..)
  * It always uses port 4587, which is **insane**. What if you have two
    webapps?
  * It requires javascript in the web browser, which
    is used to ping the server, and shut it down when the web browser closes
    (which behavior is wrong for git-annex anyway, since the daemon should
    stay running across browser closes).
  * It opens the webapp on web server startup, which is wrong for git-annex;
    instead the command `git annex webapp` will open the webapp,
    after `git annex assistant` started the web server.

Instead, I rolled my own WAI webapp laucher, that binds to any free port
on localhost, It does use `xdg-open` to launch the web browser,
like wai-handler-launch (or just `open` on OS X).

Also, I wrote my own WAI logger, which logs using System.Log.Logger,
instead of to stdout, like `runDebug` does.

----

The webapp only listens for connections from localhost, but that's
not sufficient "security". Instead, I added a secret token to
every url in the webapp, that only `git annex webapp` knows about.

But, if that token is passed to `xdg-open` on its command line,
it will be briefly visible to local attackers in the parameters of
`xdg-open`.. And if the web browser's not already running, it'll run
with it as a parameter, and be *very* visible.

So instead, I used a nasty hack. On startup, the assistant
will create a html file, readably only by the user, that redirects
the user to the real site url. Then `git annex webapp` will run
xdg-open on that file.

----

Making Yesod check the `auth=` parameter (to verify that the secret token
is right) is when using Yesod started to pay off. Yesod has a simple
`isAuthorized` method that can be overridden to do your own authentication
like this.

But Yesod really started to shine when I went to add the `auth=` parameter
to every url in the webapp. There's a `joinPath` method can can be used
to override the default url builder. And every type-safe url in the
application goes through there, so it's perfect for this.

I just had to be careful to make it not add `auth=` to the url for the
favicon, which is included in the "Permission Denied" error page. That'd be
an amusing security hole..

----

Next up: Doing some AJAX to get a dynamic view of the state of the daemon,
including currently running transfers, in the webapp. AKA stuff I've never
done before, and that, unlike all this heavy Haskell Yesod, scares me. :)


code:
[[!comment format=mdwn
 username="http://jasonwoof.com/"
 nickname="JasonWoof"
 subject="Your security solution is fine"
 date="2012-07-27T06:37:03Z"
 content="""
I think making the html file redirect is just fine. desktopcouch does something like this. They talk about it being there to help you find the port number, but it contains the security token too.

They also have some javascript to delay the redirect for a while so you can bookmark the redirect file. You can click to set a cookie to skip the delay in future.

It's been a while since I used desktopcouch, but I found an old redirect file laying around. Here it is:

http://jasonwoof.com/downloads/desktopcouch-redirect.txt

The code is pretty short, you might find something useful there.

      - Jason
"""]]


code:
The webapp now displays actual progress bars, for the actual transfers
that the assistant is making! And it's seriously shiny.

[[!img full.png]]

Yes, I used Bootstrap. I can see why so many people are using it,
that the common complaint is everything looks the same. I spent a few hours
mocking up the transfer display part of the WebApp using Bootstrap, and
arrived at something that doesn't entirely suck remarkably quickly.

The really sweet thing about Bootstrap is that when I resized my browser to
the shape of a cell phone, it magically redrew the WebApp like so:

[[!img phone.png]]

---

To update the display, the WebApp uses two techniques. On noscript
browsers, it just uses a meta refresh, which is about the best I can do. I
welcome feedback; it might be better to just have an "Update" button in
this case.

With javascript enabled, it uses long polling, done over AJAX. There are
some other options I considered, including websockets, and server-sent
events. Websockets seem too new, and while there's a WAI module supporting
server-sent events, and even an example of them in the Yesod book, the
module is not packaged for Debian yet. Anyway, long polling is the most
widely supported, so a good starting place. It seems to work fine too, I
don't really anticipate needing the more sophisticated methods.

(Incidentially, this's the first time I've ever written code that uses AJAX.)

Currently the status display is rendered in html by the web server, and
just updated into place by javascript. I like this approach since it
keeps the javascript code to a minimum and the pure haskell code to a
maximum. But who knows, I may have to switch to JSON that gets rendered by
javascript, for some reason, later on.

---

I was very happy with Yesod when I managed to factor out a
general purpose widget that adds long-polling and meta-refresh to any
other widget. I was less happy with Yesod when I tried to include
jquery on my static site and it kept serving up a truncated version of it.
Eventually worked around what's seemingly a bug in the default WAI
middleware, by disabling that middleware.

----

Also yesterday I realized there were about 30 comments stuck in moderation on
this website. I thought I had a feed of those, but obviously I didn't. I've
posted them all, and also read them all.

----

Next up is probably some cleanup of bugs and minor todos. Including
figuring out why `watch` has started to segfault on OSX when it was
working fine before.

After that, I need to build a way to block the long polling request
until the DaemonStatus and/or TransferQueue change from the version
previously displayed by the WebApp. An interesting concurrency problem..

Once I have that working, I can reduce the current 3 second delay between
refreshes to a very short delay, and the WebApp will update in
near-realtime as changes come in.


code:
[[!comment format=mdwn
 username="http://jasonwoof.com/"
 nickname="JasonWoof"
 subject="auth token length"
 date="2012-07-27T18:52:19Z"
 content="""
Your auth token looks a little short. Aren't you worried about people brute-forcing it? ;)
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="4.153.2.20"
 subject="comment 2"
 date="2012-07-27T18:55:51Z"
 content="""
Heh, I consider it overflowing most address fields a bonus, as you don't have to worry when making screenshots. :)

Of course, it changes each app run too..
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmz8d2M0lQDYWLSbDQSjYRHfrQkWKgPu60"
 nickname="Alex"
 subject="long polling in Widget form"
 date="2014-01-16T00:08:12Z"
 content="""
I've been hacking around for a couple days trying to get long polling into a Yesod web app, but I haven't found a composable solution yet. It seems like a thing worthy of generalization. Can you perhaps point me to source in git-annex that would help me take a shot at building a long polling library?
"""]]


code:
[[!comment format=mdwn
 username="http://joeyh.name/"
 ip="209.250.56.43"
 subject="comment 4"
 date="2014-01-16T03:03:48Z"
 content="""
Alex, the long polling is done almost entirely on the javascript side; the yesod handler only needs to block until the next change it wants to display. So I doubt any of what I have would make a useful library. (Although it would be very useful to have a library!)
"""]]


code:
[[!comment format=mdwn
 username="https://www.google.com/accounts/o8/id?id=AItOawmz8d2M0lQDYWLSbDQSjYRHfrQkWKgPu60"
 nickname="Alex"
 subject="re: long polling"
 date="2014-01-17T20:27:23Z"
 content="""
After poking through git-annex, I had the same realization. I hadn't realized that Yesod (or perhaps more accurately, Warp) is asynchronous above the Application level. I had feared I'd need to write some WAI middleware to accommodate the blocking nature of long polling requests. In retrospect it seems kind of silly: what kind of high-performance webserver doesn't handle requests concurrently?

I do still think there's a pattern worth encapsulating, even if it won't be long until WebSockets and SSE are widespread enough to be developed against without fallbacks.
"""]]